Tracking the progress of Language Models by extracting their underlying Knowledge Graphs
Authors: Carlos Aspillaga
Marcelo Mendoza
Alvaro Soto
Venue:  (2021)
Keywords: Language Models
NLP
Knowledge Graphs
Probe tasks
Word2Vec
GloVe
ELMo
BERT
RoBERTa
XLNet
ALBERT
T5
GPT2

Abstract: The state of the art of language models, previously dominated by pre-trained word embeddings, is now being pushed forward by large pre-trained contextual representations. This success has driven growing interest to understand what these models encode inside their inner workings. Despite this, understanding their semantic skills has been elusive, often leading to unsuccessful, non-conclusive, or contradictory results among different works. In this work, we define a probing classifier that we use to extract the underlying knowledge graph of nine of the currently most influential language models, including word embeddings, context encoders, and text generators. This probe is based on concept relatedness, grounded on WordNet. Our results show that this knowledge is present in all the models, but has several inaccuracies. Furthermore, we show that the different pre-training strategies and architectures lead to different model biases. We conduct a systematic evaluation to discover specific factors that explain why some concepts are challenging for the different families of models. We hope our insights will motivate the future development of models that capture concepts more precisely.

Under review as a conference paper at ICLR 2021
TRACKING THE PROGRESS OF LANGUAGE MODELS BY EXTRACTING THEIR UNDERLYING KNOWLEDGE GRAPHS
Anonymous authors Paper under double-blind review
ABSTRACT
The state of the art of language models, previously dominated by pre-trained word embeddings, is now being pushed forward by large pre-trained contextual repre- sentations. This success has driven growing interest to understand what these models encode inside their inner workings. Despite this, understanding their se- mantic skills has been clusive, often leading to unsuccessful, concllocles or contradictory results among different works. In this work, define a probing classifier that we use to extract the underlying knowledge graph of nine of the cur- rently most influential language models, including word embeddings, context coders, and text generators. This probe is based on concept relatedness, grounded WordNet. Our results show that this knowledge present in all the models, but has several inaccuracies. Furthermore, we show that the different pre-training strategies and architectures lead different model biases. We conduct system- atic evaluation to discover specific factors that explain why some concepts are challenging for the different families of models. We hope our insights will moti- vate the future development of models that capture concepts more precisely.
INTRODUCTION
Natural language processing (NLP) encompasses a wide variety of applications such as summariza- tion Kovaleva et al.) 2019). information retrieval (Zhan et al. 2020), and machine translation (Tang) al. 2018), among others. Currently, the use of pre-trained language models has become the de facto starting point to tackle most of these applications. The usual pipeline consists of finetuning pre-trained language model by using a discriminative learning objective to adapt the model to the requirements of each specific task. As key ingredients, these models are pre-trained using mas- sive amounts of unlabeled data that can include millions documents, and may include billions of parameters. Massive data and parameters are supplemented with suitable learning architecture, resulting in highly powerful but also complex model, whose internal operation is hard to analyze.
The success of pre-trained language models has driven the interest understand how they manage solve NLP tasks. As an example, in the case of BERT (Deylin et al.) 2019), one of the most popular pre-trained models based on Transformer architecture (Vaswani et al. [211)). several studies have attempted to access the knowledge encoded in its layers and attention heads (Tenney et al. 20 lbb Devlin et al. 2019 Hewiit& & Manning. 2019). In particular, (Jawahar et al. 2019) shows that BERT can solve tasks at a syntactic level by using Transformer blocks to encode a soft hierarchy of features different levels of abstraction. Similarly, (Hewitt Manning 2019) shows that BERT capable of encoding structural information from text. In particular, using a structural probe, they show that syntax trees are embedded linear transformation of the encodings provided by BERT.
In general, previous efforts have provided strong evidence indicating that current pre-trained lan- guage models encode complex syntactic rules, however, relevant evidence about their abilities to capture semantic information remains still elusive. As example, recent study (Si et al.) 2019) attempts to locate the encoding of semantic information as part of the top layers of Transformer architectures, however, results provide contradictory evidence. Similarly, (Kovaleva et al. 2019) fo- cuses on studying knowledge encoded by self- attention weights, however, results provide evidence for paramttreetioooooonnnn but not about language understanding capabilities.

---

Under review as a conference paper at ICLR 2021
In this work, we study to which extent pre-trained language models encode semantic information. As a key source of semantic knowledge, we focus on studying how precisely pre-trained language models encode the concept relations embedded in the conceptual taxonomy of WordNe Miiller 1995). The ability to understand, organize, and correctly use concepts is one of the most remarkable capabilities of human intelligence (Lake al. 2017). Therefore, a quantification of the ability that a pre-trained language model can exhibit to encode the conceptual organization behind WordNet is highly valuable. In particular, it can provide useful insights about the inner mechanisms that these models use to encode semantic information. Furthermore, an analysis of concepts and associations that result difficult to these models can provide relevant insights about how to improve them.
In contrast to most previous works, we do not focus on particular model, but target a large list of the most popular pre-trained language and text-embedding models. this sense, one of our goals is to provide a comparative analysis of the capacities of different types of approaches. Following Hewitt Manning (2019), we study semantic performance by defining a probing classifier based on concept relatedness according to WordNet. Using this tool, we analyze the different models, enlightening how and where semantic knowledge is encoded. Furthermore, we explore how these models encode suitable information to recreate the structure of WordNet. Among our main results, we show that the different pre-training strategies and architectures lead to different model biases. In partieular, we show that contextualized word embeddings, such BERT, encode high-level con- cepts and hierarchical relationships among them, creating a taxonomy. This finding corroborates previous work results (Reif et al., 2019) that claim that BERT vectors are stored in sub-spaces that have correspondence with semantic knowledge. Our study also shows evidence about the limita- tions of current pre-trained language models, demonstrating that they all have difficulties encoding specific concepts. As an example, all the models struggle with concepts related to "taxonomical groups", performing worse than chance in some cases. Our results also reveal that models have very distinctive patterns in terms of where they encode most of the semantic information. These patterns are dependant on architecture and not on model sizes.
2 RELATED WORK
The success of deep learning architectures in various NLP tasks has fueled a growing interest from the community to improve understanding of what these models encode. Several works have studied these models' impact on downstream tasks at the syntactic or semantic level. Some studies (Tenney) ltt al. 2019b) claim that success in specific task helps understand what type of information the model encodes. Other studies have improved the understanding of what and where these models en- code information, by analyzing correlations between such as layers (Jawahar et al. 2019). encoded hidden states (Tang et al., 2018 2019), and attention heads (Michel et al. 2019).
and specific architecture blocks,
Saphra & Lopez,
Evidence of syntactic information: Using probing classifiers, Clark et al. (2019) claims that some specific BERT's attention heads show correspondence with syntactic tasks. Goldberg (0019) il- lustrates the capabilities that BERT has to solve syntactic tasks, such as subject-verb agreement. BERT's s success in these tasks fuels the belief that BERT can code the syntax of a language. Hewitt & Manning (2019) proposes structural probe that evaluates whether syntax trees are encoded in a linear transformation of BERT embeddings. The study shows that such transformation exists in BERT, providing evidence that syntax trees are implicitly embedded in BERT's vector geometry. Reif et al. (2019) has found evidence of syntactic representation in BERT's attention matrices, with specific directions in space representing particular dependency relations.
Evidence of semantic information: Reif et al. (2019) suggests that BERT's internal geometry may be broken into multiple linear subspaces, with separate spaces for different syntactic and semantic information. Despite this, previous work has not yet reached consensus about this topic. While some studies show satisfactory results tasks such as entity types (Tenney et al. 2019a). semantic roles (Rogers et al. (020). and sentence completion (Ettinger] (020)), other studies show less favor- able results in coreference (Tenney et al.] (019b) and Multiple-Choic
Reading Comprehension (Si) et al. 2219), claiming that BERT's performance may not refleet the model's 's truc ability of language understanding and reasoning. Some works have studied which blocks of BERT are used to solve
'WordNet is human-generated graph, where each one of its 11 117000 nodes (also called synsets) represent concept. In this work use the hyponymy relations, which represent if concept subelass of another.
2

---

Under review as a conference paper at ICLR 2021
tasks at the semantic level. Tenney et al. (2019b) proposes set of edge probing tasks to test the encoded sentential structure of contextualized word embeddings. The study shows evidence that the improvements that BERT and GPT-2 offer over contextualized embeddings Glo is only significant in syntactic-level tasks. Regarding static word emheddings, Yaghoobraeee et al. (2019) shows that senses well represented in single-vector embeddings if they are frequent, and that this does not have negative impact on NLP tasks whose performance depends on frequent senses.
Layer-wise or head-wise information: Tenney et al. (2019a) shows that the first layers of BERT focus on encoding short dependency relationships at the syntactic level (e.g.. subject-verb agree- ment) while top layers focus on encoding long-range dependencies (e.g.. subject-object dependen- cies). Peters et a0.1 8a) supports similar declarations for Convolutional. LSTM, and self-attention architectures. While these studies also support that the top layers appear to encode semantic infor- mation, the evidence to support this claim is not conclusive or contradictory with other works. As an example, Dawahar et al. (2019) could only identify one SentEval semantic task that topped the last layer. In terms of information flow, Voita et al. (2019a) reports that information about the past in left-to-right language models gets vanished as the information flows from bottom to top BERT's s lay- ers. |Hoo et al. 2019) shows that the lower layers of BERT change less during finetuning, suggesting that layers close to inputs learn more transferable language representations. In terms of architecture design. et al. (2020) provides evidence that increasing self-attention the bottom and increas- ing feed-forward sub-layers at the top improves results in language modeling tasks using BERT. Other studies have focused on understanding how self- heads contribute to solving specific tasks (Vig 2019). Kovaleva et al. (2019) shows set of attention patterns that is repeated across different heads when trying to solve GLUE tasks (Wang al. 2018). Furthermore, Michel et al. (201)) and Voita et al. (2019b) show that several heads can be removed from trained Transformer models without degradation in downstream tasks.
In summary, on the one hand, related work shows results that provide strong evidence conclud- ing that BERT and other Transformer-based models can encode information at the syntactic level. Furthermore, BERT uses this information to solve various benchmark NL.P tasks. Working with ambiguous words has allowed BERT to perform well on Machine Translation tasks (MT), and out- performing other architectures in word sense disambiguation (WSD). On the other hand, while some studies claim that top layers are helpful to solve semantic tasks, the results are not conclusive. Ef- forts to locate which blocks of the Transformer architecture operate the semantic level have been unsuccessful. In this context, our work helps to fill the gap between the deeper understanding that have now about how word embeddings and language models encode and work with syntax, and the still shallow comprehension of their abilities to encode and work with semantics.
3 STUDY METHODOLOGY
Probing methods consist of using the representation of a frozen pre-trained model to train a classifier to achieve a particular task. If the probing classifier succeeds in this setting but fails using an alternative model, it means that the source model encodes the knowledge needed to solve the task. Furthermore, the performance of the classifier can be used to measure how well the model captures this knowledge (Conneau et al. 2018). Following Hewitt & Manning (2019), we use similar probing method at the semantic level and apply it to the nine models presented in Section B.I Probing techniques have shown useful for NLP tasks (Saphra & Lopez, 2019). however, they have also been questioned (Ravichander et al., 2020). as probe success does not guarantee that the model relies on that information to solve target task. Consequently, our study limits to shed light on whether the models under evaluation encode relevant knowledge to solve the task of predicting concept relatedness in semantic ontology such Wordnet.
To study how precisely the models encode semantic information, we measure correctness in pre- dicted relations among concepts at two levels: (a) pair- wise-level by studying performance across sampled pairs of related or unrelated concepts, and (b) graph-level by using pair-wise predictions to reconstruct the actual graph. We describe these two approaches in Sections 3.2 and 3.3 respectively. They essentially study the same phenomenon as both share the exact same model predictions. They only differ in the way they use these predictions.
3

---

Under review as a conference paper at ICLR 2021
3.1 WORD EMBEDDINGS MODELS
This study considers the most influential language models from recent years. We consider the essen- tial approaches of three model families: non contextualized word embeddings (NCE), contextualized word embeddings (CE), and generative language models (GLM). We consider Word2Vec (Mikolov ctt al. 2 and Glo (Pennington et al., 2014) for the first family of approaches. For the CE family, consider ELMo (Peters et al.) 2018b), which is implemented
bidirectional LSTM architecture, XLNet (Yang et al. 2019), and BERT (Devlin et al. (019) and its extensions ALBERT
(Lan 2020) and RoBERTa Liu al.] 2019), all of them based on the Transformer architecture. GPT-2 (Radford et al., 2018) and T5 (Raffel et al. 2019) are included in the study to incorporate approaches based on generative language models.
3.2 SEMANTIC PROBING CLASSIFIER
We define edge probing classifier that learns to identify if two concepts are semantically re- lated. To create the probing classifier, we retrieve all the glosses from the Princeton WordNet Gloss Corpu This dataset provides WordNet's synsets gloss sentences with annotations identifying oc- currences of concepts within different sentence contexts. The annotations provide a mapping of the used words to their corresponding WordNet node. We sample hypernym pairs A, B. Then, from an unrelated section of the taxonomy, we randomly sample a third synset C, taking care that C is not related either A or B. Then, (A, B, C*) forms triplet that allows us create six testing edges for our classifier. To train the probing classifier, we define a labeled edge {a, y, L}, with r and synsets in B, C), F y. L E {0, is the target of the edge. If y direct or indirect parent of x. L = I, while L = 0 in other case. For each synset 2, we sample one of its sentences S(r), (v) from the dataset. Let M be word embedding model. If M belongs the NCE family, x and are encoded by (x) and My). respectively. If M belongs to the CE or GLM families, then x and y are encoded by the corresponding token of M (xx)) and M (S())), respectively. Accordingly, the context of each concept provides additional information for the creation of its word embedding.
P
Concept
Sampled senteacca concepts are used
Embeldiigg
Pre-trained Model
WordNei supervision
>>
Linear Layer
MLP
"
"
Mly)
Mix)
Figure 1: Inputs to the edge probing classifier correspond to the model embeddings Mxx) and M(y) of concepts x and respectively. To account for the different dimensionality of the tested models, (x) and (y) are projected into a common lower dimensionality space using linear layer. The resulting embeddings z' and y' are concatenated into vector, which is fed into Multi-Layer Perceptron that is in charge of predicting if the concept pair is related or not.
first To accommodate for the evaluation of concept embeddings of different dimensionalities, project the embeddings each concept æ and y into a low dimensionality space using a linear layer (see Figure []. Choosing a low target dimensionality for this linear layer was also essential to avoid overfitting the classification module. Then, these vectors, denoted as x' and y', are concatenated and fed into Multi-Layer Perceptron (MLP) classifier. The linear layer and the MLP are the only trainable parameters of our setting, as we use the source model weights without any finetuning.
Similar structural probes have been proposed to evaluate syntax encoding in BERT, However, a MLP to learn the structural relation between concept pairs, providing the test with a mechanism that allows the embeddings to be combined in a non-linear way. Tests based linear transforma- tions such as the one proposed by Hewitt& & Manning (2019) did not allow us to recover the WordNet
4

---

Under review as a conference paper at ICLR 2021
structure. This indicates that the sub-spaces where the language models encode this semantic infor- mation are not linear. The fact that syntactic information is linearly available with high accuracy, makes us believe that syntax trees might be critical intermediate result needed for the language modeling task; whereas semantic information emerges as an indirect consequence accurate lan- guage modeling and syntactic identification, but might not constitute information that the model relies on for the language modeling task, as postulated by RRavichander et al. (2020).
3.3 RECONSTRUCTING THE STRUCTURE OF A KNOWLEDGE ARAPH
The proposed probe classifier predicts if pair of concepts (u, v) form a valid (parent, child) rela- tion according to WordNet, where (u,,)) E [0, 1] denotes the corresponding classifier output. It is important to note that valid (parent, child) relations include direct relations, as in (dog, poodle), and transitive relations, as in (animal, poodle). and that the order of the items in the relation do matter.
To reconstruct the underlying knowledge graph, for each valid (parent, child) relation given by (u threshold, we need an estimation of how close are the corresponding nodes in the graph. To do this, we introduce the concept of "parent closeness" between parent node u and child node v, that we define as de (u, v)) We propose two alternative scores to estimate de:
i) Transitive Intersections Metric (TIM):
we explore a metric grounded directly in the tree struc- ture of a knowledge graph. The key observations are that nodes и and that form a parent-child relation have some transitive connections in common. Specifically, all descendants of are also descendants of u, and all the ancestors of u are also ancestors of v. Furthermore, the closer the link between and in the graph, the bigger the intersection. With that in mind, for each candidate edge e = (u, v) we define the following metric for de:
(u, v) = -
Σ JENMuee
h( +
Σ JEN
(ias) * hiu.o)
(1)
where the first sum accounts for the similarity in the descendants of nodes and v, and the second sum accounts for the similarity in the ancestors of nodes u and The term h(a.)) at the right-hand
side accounts for the direction of the edge. Constant C maxur,y d(u,, Uz) is added force de to be positive. N denotes the set of all nodes (concepts).
ii) Model Confidence Metric (MCM): All nine models in this study capture close concept rela- tions more precisely than distant relations (supporting evidence can be found in Appendix ( In practice, this means that concept like poodle will be matched with its direct parent node dog with higher confidence than with a more distant parent node such as animal. Thus it is reasonable to use do, v) = [(,e))-
Based on de, natural choice to find the tree that comprises only the closest parent of each node to obtain the minimum
analogous to the minimum-spanninn--an aaaaning--rboreseenne (MSA) of the graph defined by de. The MSA is (MST) objective used in Hewitt & Manning (2019). but for the specific case of directed graphs. For mathematical formulation of the MSA optimization problem applied our setting, refer readers to Appendix A.3
4 HOW ACCURATE IS THE SEMANTIC KNOWLEDGE?
4.1 SEMANTIC EDGE PROBING CLASSIFIER RESULTS
Table the results obtained using the edge probing classifier in several model variants. Results show that regardless of model sizes, performance is homogeneous within each family of models. Additionally, results show that when all the layers are used, NCE and GL.M methods obtain a worse performance than those achieved by CE methods. When single layers are used, GLM show improved performance, suggesting that the generative objective forces the model to capture semantics earlier in the architecture in order to keep the last layers for senerative-specifee purposes that do not have a direct relationship with this kind of knowledge. In contrast, CE models degrade or maintain their performance when single layers are used.
Notice that results in Table T show pair-wise metrics not graph metrics. As we are dealing with graphs, predicted edges built upon related predicted edges, thus drifts in small regions of the
5

---

Under review as a conference paper at ICLR 2021
Emb. Size Al//Best Layer
Best Layer
F1-score All Layers
Fl-score
Family
Model
Best Layer
Vord2Vec Mikolov et al
2013
300/-
.7683 + .135
NCE
VloBe-42B ( Peanincton
2014
300 /-
.7877 + .0844
GPT-2 (Radfodd et
2018
9984 768
.7862 + 0132
.7921 + 0108
T5-small Ralle
7168 512
.8156
.8999 +
GLM
GPT2-xl
78400 1600
13
.7946
.0299 1
T5-large Raffel 2019)
51200 1024
8148 + .119
.8331 + 0102
El.Mo-smal (Peters al
20186
256
.7986 1. .0126
.7800 1 .1119
BERT-hase Dev
2019
9984 768
.2240 + .023
.8185 + .104
RoBERTa-base base
Lau et
2019
9984 768
5
.392 + .0100
.2266 + .0033
XLNet-base Yang er
2019)
9984 768
4
.306 + .113
.2993 + 16
ALBERT-base (Lan
2070
9984 / 768
12
.1884 + 0222
.0733 + .102
CE BERT-large Deviin
ELMo-large Peters ol
20136
3072 / 1024
2
8311 + 00900
.3300 = .0833
2019)
25600 / 1024
14
.8178 1. .052
.8185 1 .113
RoBERTa large
25600 1024
13
.219 +
.814 + 00822
XI-Net-lare Yang et
25600) 1024
.8211 1: .8244 +
ALBBERT-xxlarge
2020
53248 4096
.8333
.8194 +
Table 1: Results obtained using the edge probing classifier. We study the performance in many
model's 's variants, considering small and large versions of several models. Results are shown grouped by the families of methods defined in Section 3.1
graph may cause large drifts in downstream connections. Furthermore, our setup balances positive (links) and negative samples (not links), but the proportion of negative samples will be considerably larger in a real reconstruction scenario. As a consequence, we emphasize that these numbers must not be considered in isolation, but together with the results reported in sections 4.2 and
4.2 EXTRACTING THE UNDERLYING KNOWLEDGE GRAPH
Predicting a knowledge graph has a complexity of at least O(N²) in the number of analyzed con- cepts. In our case, this imposes highly demanding computational obstacle because WordNet has over 82000 noun synsets. To accelerate experimentation and to facilitate our analysis and visualiza- tions, we focus on extracting WordNet sub- sub-graph that comprises only 46 nodes. These nodes are picked to include easily recognizable relations. We use the tree-edit-distance to evaluate how close are the reconstructed graphs to the target graph extracted from WordNet. Table 2 shows our results.
Balm pine spinach
Sxpnnach
chicken
lettuce
ganary
Burtte
herb
plant
weledy
bing diper
parrer
Family
Model
TIM
Tree Edit Dist.
MCM
Avg.
@ snake
amigiiiian
animal
livin, ting
edjeecz
ubsttaccion
appportnntn rpprortimatiii
NCE
GloVe-42B
Word2Vec
56
59
59
58
59
hame Saplems
fedine
ammummelll
Taaturel objenc
Meriity
GLM
GPT-2
53
57
55
qoality criernsss
T5
58
55
56
primate
aligater
attripute
narnenesss
Aimilaitty
ELMo
52
crocodile
BERT
49
48
49
mmonkey
e plabyous
CE
RoBERTa
56
55
cryctaal
Squaiity
XLNet
52
48
50
regatle
calcutie
ALBERT
53
50
51
reconstructed graph using MCM over BaRT-large. The graphs for all the models can be found in Appendix
Figure 2:
Table Tree Edition Distance against the ground truth graph (large models used). We display both strategies for estimating de along with their average score.
Table shows that the knowledge graphs retrieved using CE models are closer to the target than graphs provided by NCE and GLM models. In particular, the best results achieved by BERT, ALBERT, and XLNet, indicating that these models encode more accurate semantic information than the alternative models. These results are consistent with those obtained in Section 4.1

---

Under review as a conference paper at ICLR 2021
Visual inspection of the reconstructed graphs the one found in reeuals that models capture some key categories but fail to map some finegrrinedd relations.
5 WHAT IS EASY OR HARD? WHAT ARE THESE MODELS LEARNING?
Section shows that different model families differ in their emrors. Furthermore, it shows that within the same family, models have similar biases. In this section, we elucidate which semantic factors have an impact on the performance of these models and which ones do affect their Fl-score.
Medan FL 21-cc
Werzzees
WontZre:
WertaVes
OF72
cart
To-bige
3237-lare
BOiT large
BEXT
Rast luree
Ruseeta lore
AdBiif-acaggg
(a)
(b)
(c)
Vin
Xo
109%
12
25
15
10:003
6.0
6:0
Ulow-423 423
6::
pleve
GFT2
Thiagg
arga
st4l-targe
san1 largt
orga
AURIIT adppp:
ALDERTiolirge
ALEERT-olarpe
Figure 3: Semantic factors with high (top charts) or low (bottom charts) impact on F1 Fi-score, along with their 90% confidence intervals. Charts only display ranges where at least 100 samples existed. Appendix C shows additional factors along with the specific implementation details.
Figure a shows that most models decrease their F1-score as concepts get more specific (increased depth). We hypothesize that this because higher-level concepts (e.g. Animal) appear more fre- quently and in more diverse contexts, as they are also Dog, Cat, Chihuahua), allowing the models to learn more precise representations for them. In contrast, lower-level concepts will only be present in their specific contexts (e.g. texts about Apple- Head-Clin Chihuahua). This hypothesis is confirmed by Figure b,, as concepts with higher number of sub-classes have higher Fl-scores. Furthermore, Figure 3 also shows that models tend capture concepts more accurately as the model is exposed to them more frequently in the pre-training corpus.
instances of their subclasses (e.g.
Additionally, Figure 3 indicates that the models degrade their F1-score when concepts are frequent. In particular, NCE and GLM models are more sensitive to this factor. We hypothesize that this is because high-frequency concepts appear in contexts with too much variability. Their unspecific use makes more difficult to distinguish accurate meaning for NCE and GLM models.
Another surprising finding that CE and GLM models are almost unaffected by the number of senses that certain word has, neither to their sense ranking their number of sibling concepts, displaying almost flat charts (Figure d-e-f.. This finding suggests that these models pay as much attention to the context as to the target word, and are probably biased in favor of contextual informa- tion. This intuitive for Masked-I Langodd-Lnggggge--a Models Models such as BERT, but not for others, such as GPT-2. Furthermore, this behavior is opposed to what NCE models exhibit according to Yaghoobzadeh et al. 2019)) and our experiments, as NCE models tend to focus more on the most frequent senses.
Analysis of specific semantic categories indicates that in most cases, the models of the same family have similar behaviors, especially within the NCE or CE families. More importantly, different fami- lies show different patterns. Table TableBishor shows some salient examples. Surprisingly, all models struggle with the category "taxonomic groups". Manual inspection sentences and analysis of related con- cepts make us believe that in these cases the use of the context confuses CE and GLM models, in

---

Under review as a conference paper at ICLR 2021
living thing
social
taxonomic
Family
Model
artifact
attribute
matter
person
group
group
Word2Vec
.7120
.7044
.7295
.7022
.7208
.7497
.6200
NCE
GloVe-42B
.7389
.7213
.7421
.7633
.7351
.7579
.6648
GPT-2
.7903
.7730
.7300
.7582
.7207
.8155
.3030
GLM
T5
.7868
.6499
.7862
8002
.7735
.7868
.6944
ML
.8308
.8093
.8187
.7756
.8022
.8312
BERT
.8249
.8094
.7593
.7645
.7379
.8516
.4804
RoBERTa
.8315
8167
.7823
.7614
.7585
.4921
CE
XLNet .8319
.0644
.7907
.7636
.7779
.8422
.5371
ALBERT
.8231
8050
.7758
.7685
.7826
.8556
.4277
Table Average FI -core for some semantic categories revealing models strengths and weaknesses. Several other categories reported in Appendix G along with their standard deviations.
many sentences the corresponding concept could be nicely replaced by another, conveying a modi- fied but still valid message. This phenomenon does not take place in other categories such "social group" or "attribute", even though these concepts are closely related to "taxonomic groups".
WHERE IS THIS KNOWLEDGE LOCATED?
As mentioned in Section 2 prior work has not shown consensus about where semantic information primarily encoded inside these architectures. Our experiments shed light on this subject. Figure shows how each layer contributes to the Fl-score.
ELNO
EUMo large
BERT-bos base
XaNen-base
GPT2
BERT
XLNer-large
GPT2-XL
RuDERTa-bese
ALSERT-ban
T5-mmall
(a)
(b)
(c)
Hidden 2ww.2o20a
peis maiiiiis20000eeeeee Hidden Layer
Layer Index
Figure 4: F1-score for hypernym prediction across each model layer.
Figures D 4Hb show the performance across layers for the CE-based models. They reveal that while BERT and RoBERTa use their top-layers to encode semantic information, XLNet and AL- BERT use the first layers. Figure Te shows that GLM models have very distinct patterns. While GPT-2 encodes semantic information in all the layers, T5 shows M shape because its configu- ration considers an encoder-decodee architecture. The chart shows that T5 uses its encoder to hold most of the semantic information. We also note that small models show similar patterns as their larger counterparts, suggesting that these patterns depend on the architecture and not model size.
7 FURTHER DISCUSSION AND IMPLICATIONS
summariee our main findings. Findings (1), (2) and (3) indicate that, to a different extent, all models encode relevant knowledge about the hierarchical semantic relations included in WordNet. However, some of this knowledge is incorrect. In particular, as we mention in Section we observe that the ability to learn about concept depends on its frequency of appearances the training corpus and the level of specificity of its meaning. Furthermore, some concept categories result difficult for all model families, while some categories are particularly difficult for contextual models such as CE. We hypothesize that stronger inductive biases are required to capture low-frequency concepts. Furthermore, we believe that new learning approaches are needed to discriminate accurate meaning for high-free frequency concepts. As expected, our findings indicate that model families have different biases leading to different behaviors. Thus, our results can illuminate further research to improve semantic capabilities by combining the strengths of each family of models. As example,
8

---

Under review as a conference paper at ICLR 2021
Supporting
Finding
Evidence
Involved Models
(1) All models encode a relevant amount of knowledge about semantic All
All
relations in WordNet, but this knowledge contains imprecisions.
The ability to learn concept relations depends on how frequent and Fig.
3
NCE and GLM
specific are the concepts, affecting some model families heavier.
(3) Concept difficulty is usually homogeneous within each model fam- Table
All
ily. Some semantic categories challenge all models.
Some models encode stronger semantic knowledge than others, Tables usually according to their family.
ML, BERT, RoBERTa, ALBERT, XLNet. T5
(5) Some models focus their encoding of semantic knowledge in spe- Table
Fig.
GLM
cific layers, and not distributed across all layers.
(6) Models have distinctive patterns to where they encode semantic Table
Fig.
AlI
knowledge. Patterns model-specific and size-specifie.
Model size has impact in the quality of the captured semantic knowledge, seen in layer-level probe tests.
Table
Fig.
ELMo, RoBERTa, AL BERT, GPT-2, T5
Semantic knowledge docs not depend on pre-training corpus size.
Tables
B
Contextual models are unaffected by multi-sense words.
Fig.
CE and GLM
Table 4: Summary of our main findings and their corresponding supporting evidence.
one could combine them ensembles or by incorporating more than single model head, each one equipped with a different loss function (i.e., one generative approach resembling GLM-based methods and another discriminative resembling CE-based methods).
Findings (4), (5) and (6) suggest that instead of standard finetuning of all layers of BERT according to given downstream task, to improve semantic capabilities one could perform task profiling to decide the best architecture for the task and also how to take advantage of it. By using only a limited number of layers choosing different learning rate for each layer, one could exploit the semantic knowledge that the pre-trained model carries, avoiding the degradation of this information present the top layers, especially when using T5, XLNet, or ALBERT-large. Accordingly, recent work on adaptive strategies to output predictions using a limited number of layers (Xin et al. 2020 Liu et al.,
2020 Hou al. 2020 Schwwrtz et al., 2020: Fan et al. 2020 [Bapaa et al.) (2200) would benefit from using architectures that encode knowledge in the first layers. To the best of our knowledge, these works have only used BERT and RoBERTa, achieving a good trade-off between accuracy and efficiency. Only Zhou et al. (2020) has explored ALBERT, reporting improved accuracy by stopping earlier. Our findings explain this behavior and suggest that T5 or XL.Net may boot their results even further these architectures have sharper and higher information peaks in their first layers.
Findings (7) and (8) suggest that recent success in semantic NLP tasks might be due more to the use of larger models than large corpora for pretraining. This also suggests that to improve model perfor- mance in semantic tasks, one could train larger models even without increasing the corpus size. A similar claim has been proposed by Li et al. (022)) leading empirical performance improvements.
Finally, finding (9) is important because it suggests that contextual models pay as much attention to the context as the target word, and are probably biased in favor of contextual information, even they are not based on the Masked-Languuage- Language-Moode setting. We believe that this inductive bias could be exploited even further in the design of the underlying architecture, thus this finding might elucidate design direction to encourage more effective learning of semantic knowledge.
8 CONCLUSIONS
In this work, we exploit the semantic conceptual taxonomy behind WordNet to test the abilities of current families of pre-trained language models to learn semantic knowledge from massive sources of unlabeled data. Our main conclusion is that, indeed, to significant extent, these models learn relevant knowledge about the organization of concepts in WordNet, but also contain several impre- cisions. We also notice that different families of models present dissimilar behavior, suggesting the encoding of different biases. As part of future work, we hope our study helps to inspire new ideas to improve the semantic learning abilities of current pre-trained language models.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Controlling computation versus quality for neural sequence models. CoRR, abs/2002.07 106, 2020. URL
org. abs/
1002.07100
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT"s attention. CoRR, hta/1996.043411, 2019.
Alexis Comneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can eram into a single S&!#* vector: Probing sentence embeddings for linguistic proper- ties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers). pp. 2126-2136, Melbourne, Australia, July 2018, Association for Computational Linguistics. doi: 10.18653/v 1/P18- 117118--198 1198. URLhttps:: http://wwwwwwww..bb....b. aclweb. org/ aanhology/P133 /P18 1198
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pp. 4171 4186, 2019.
Allyson Ettinger. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the. Association of Computational Linguistics, TACL, 8:34 48 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2020. URL hnttps://openrewe .nt meet/foruumldd-yy022y5.
Yoav Goldberg, Assessing BERT's syntactic abilities. CoRR,
2019.
Yaru Hao, Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pp. 4141-4150, 2019.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIT 2019, pp. 4129- 4138, 2019.
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems, 2020.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? Proceedings of the 57th Conference of the Association for Computational Linguis- fics, ACI. 2019, pp. 3651-3657, 2019.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, EEMNL--JCNNL 2019, pp- 4365-4374, 2019.
B. Lake, T. Ullman. J. Tenenbaum, and S. Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman. Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: lite BERT for self-supervised learning of language representations. In 8th International Conference Learning Representations, ICI. 2020, 2020.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, K. Keutzer, D. Klein, and J. Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. ArXiv, abs/2002.11794 2020.
10

---

Under review as a conference paper at ICLR 2021
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. FastBERT: self- distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6035-6044, Online, July 2020. Association for Computational Linguistics. doi: 10.1 aaa0://-ii.220...mmmma.3.333335 URL https://www.
acl celweb..o org/ antho ogy/ wttp///ttooolyyy////// acc mammmmmin... 537
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. CoRR, Abs/1907.1 16692, 2019.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pp. 14014-14024, 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed rep- resentations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, NIPS 2013, pp. 3111-3119, 2013.
George A. Miller. WordNet: A lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2014, pp. 1532-1543, 2014.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 1509, Brussels, Belgium, October- November 2018a. 8a. Association for Computational Linguistics. doi: 10. 10.18853/11/D1--1 1179. URL https:: www.w//wwww..//mm ac web orgg antho ogy/ DD8 1179
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2018, pp. 2227-2237, 2018b.
Ofir Press, Noah A. Smith, and Omer Levy. Improving transformer models by reordering their sublayers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pp. 2996-3005, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2018. URL
maagueggeeeggg.aaemmmme....... pdd
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2019.
Abhilasha Ravichander, Yonatan Belinkov, and F. Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? ArXiv, abs/2005.00719, 2020. URL https::/://axii
org/abs/2005. 00719
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viégas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pp. 8592-8600, 2019.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. primer in BERTology: What we know about how BERT works. CoRR abs/2002. 12327, 2020.
Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with sveca. In NAACL-HLT (1), pp. 3257-3267, 2019. URL lattps://acll wwtg/nthhooogg..ooooooooooooooot
papers/N/NI 9 9-
11

---

Under review as a conference paper at ICLR 2021
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. The right tool for the job: Matching model and instance complexities. In Proc. of ACL, 2020.
Chenglei Si, Shuohang Wang, Min-Yen Kan, and Jing Jiang. What does BERT learn from multiple- choice reading comprehension datasets? CoRR, 0ts/1910.12399 2019.
Gongbo Tang, Rico Sennrich, and Joakim Nivre. An analysis of attention mechanisms: The case of word sense disambiguation in neural machine translation. In Proceedings of the Third Conference on Machine Translation, WMT 2018, pp. 26-35, 2018.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pp. 4593 4601, 2019a.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, 2019b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems NIPS 2017, pp. 5998-6008, 2017.
Jesse Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pp. 37-42, 2019.
Elena Voita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the trans- former: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019 pp. 4395-4405, 2019a.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Conference of the Association for Computational Linguisties, ACL 2019, pp. 5797-5808, 2019b.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, Black- boxNLP@ EMNLP 2018. pp. 353-355, 2018.
Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2246-2251, Online, July 2020. Association for Computational Linguisties. URL httpt::t/// /www acwweb org/ ant hwtgg//nnnooogy//00000
acl main. 204
Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, and Hinrich Schütze. Probing for semantic classes: Diagnosing the meaning content of word embeddings. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 5740 5753, Association for Computational Linguistics, 2019. doi: 10. 10.18653/v1/p19-155 1574.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. In Anmual Confer- ence on Neural Information Processing Systems 2019, NeurIPS 2019, pp. 5754-5764. 2019.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. An analysis of BERT in
document ranking. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, pp. 1941-1944, 2020.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses pa- tience: Fast and robust inference with early exit, 2020.
12

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
A.1 EDGE PROBING CLASSIFIER DETAILS
To study the extent to which these Language Models deal with semantic knowledge, we extend the methodology introduced by Tenney al. (2019b). In that study, the authors defined a probing classifier at the sentence level, training supervised classifier with task- label. The probing classifier's motivation consists of verifying when the sentence's encoding help to solve a specific task, quantifying these results for different word embeddings models. We cast this methodology to deal with semantic knowledge extracted from WordNet. Rather than working at the sentence level, define an edge probing classifier that learns to identify if two concepts are semantically related.
To create the probing classifier, we retrieve all the glosses from the Princeton WordNet Gloss Cor- pus The dataset provides WordNet's synsets gloss with manually matched words identifying the context-appron appropritt sense (see Figure 5). In WordNet, each sense is coded as one of the synsets related to the concept (e.g., sense tendency.n.03 for the word tendency). Using synset A and its specific sense provided by the tagged gloss, we retrieve from WordNet one of its direct or indirect hypernyms, denoted B (see Figure (6) If WordNet defines two or more hypernyms for A, we choose one of them at random. We sample a third synset C, at random from an unrelated section of the taxonomy, taking care that Cs not related to either A B (c.g., animal, OI). Then, (A, B, C") form triplet that allows us to create six testing edges for our classifier: (A, B), which is com- pounded by a pair of related words through the semantic relation hypernym of, and five pairs of unrelated words ((A, C), (B, C), (B,A), (C, A). (C, B)). We associate label to each of these pairs that show whether the pair related or not (see Figure Note that we define directed edges, meaning that the pair (A, B) is related, but (B, A} is unrelated to the relationship hypernym of. Accordingly, the edge probing classifier will need to identify the pair's components and the order in which the concepts were declared in the pair.
Gloss: "Having power or capacity or tendency to absorb or soak something"
WordiNet tag:
tendency.n.03
Figure 5: The Princeton WordNet Gloss Corpus provides sentences with manually annotated map- pings of words to their corresponding WordNet Synset (concept / sense).
A
<1>
B disposhioo.n. n.
- <0>
hypernym
hypernyn
B
7 <0>
hypormyn
not hyppamym
B - <0>
+ <
A landency..05
not hypomym
animal n.01
<0>
plccied
randoo
concept pairs
labols
Figure 6: Each triplet is used to create related and unrelated pairs of words according to the relation- ship hypernym of. We create six edge probing pairs, and therefore, the edge probing classifier will need to identify the pair's components and the order in which the words were declared in the pair.
We create training and testing partitions ensuring that each partition has the same proportion of leaves versus internal nodes. The latter is essential to identify related pairs. During training, we guarantee that each training synset is seen at least once by the probing classifier. To guarantee the above, we sample each synset in the training set and sample some of its hypernyms at random. Then. we randomly sample some unrelated synset for each related pair that has no relation any
httpss:/weerdhettodd...r..
13

---

Under review as a conference paper at ICLR 2021
of the words in the related pair. We create three partitions from this data on 70/15/15 for training, development, and testing foldings, respectively.
We train the MLP classifier using weighted binary cross-entropy loss function. Since have one positive and five negative examples per triplet, we use a weighted loss function with weights 5 and for the positive and negative class, respectively. Accordingly, positive and negative examples have the same relevance during training. We implemented the linear layer and the MLP classifier using a feed forward network with 384 hidden units. The MLP trained using dropout 0,425 and L2 regularizer to avoid overfitting.
To create the vector representations for each of the word embeddings models considered in this study, we concatenate the hidden state vectors of all the layers for each tagged synset. For both CE and GLM-based models, each gloss was used as context to build specific contextual word embeddings. If the gloss has more than one tagged token, we take only the first of them for the analysis.
A.2 WORDNET METRICS: DISTANCE
Formally, let dw (x, be the Wordnet distance between two synsets x, y, defined by:
dw (1, v) Aputt (I,
if y is ancestor of x,
dpatts (I, 2) + duatt (y: 2)
otherwise,
where dpuih (I. y) is the length the shortest path between and y in WordNet, measured number of hops, and 2 is the closest common ancestor of x and in the case that is not ancestor of a.
A.3
OPTIMIZATION PROBLEM APPLIED TO KNOWLEDGE GRAPH RECONSTRUCTION
Given graph G' with nodes N and unknown edges E, define auxiliary graph G' with nodes N and edges E', comprised of all possible directed edges. For each edge E E', we obtain a prediction he that estimates the probability of that edge representing valid hypernymy relation. and distance de that estimates the "parent closeness" between the nodes in G.
We define 8(v) to be the of edges {(u, v) (parent, child) relation. We also define (SS) to be the set of edges {(u, v) E E" u S, S}. We estimate the graph topology of G defined by E" by solving the following optimization problem:
N,u u * v} where edge (u, represents
Σxche sc BEF"
PEN
Ie E {0,1]
e L' W (r) AS cS
s.t. Ze E arg min
Σ oE Ey
Iede
s.t.
Σ EeE(() Te =
(3)
EeEr(S) Ze 2
Objective function 2 is used to find the best root node r; and the nested optimization problem is the minimum spanning arborescence problem applied to the dense graph G". The final binary values of ze estimate E by indicating if every possible edge exist in the graph or not. To solve this optimization problem, we need estimates of he and de for each edge We use the output of the probing classifier as an estimate of the probability of he, and use TIM and MCM scores as estimates for d, (See Section 3.3
"The value of this distance will be small if the hypernym relation close, large if it distant valid.
14

---

Under review as a conference paper at ICLR 2021
B RECONSTRUCTED GRAPHS
salamender
monbey tiper @hemi sapient cat monotreites mammal
frog smiphibian
teline
pine wondy plant paim
platypuc primate
living thing
plant
herr lettuce
aanimal
spinach
snake
ottlect
entity
abstraction
turple
mpliee
attribute
Gamenies likeness
similanit
Procialle
Ideenty approxtimation
quality
sligater
natural object
oneeees
puirot
bird
chicken
cansry
crystal caicullas
rock
equaliity
Figure 7: Ground Truth Knowledge Graph
paim
pine
feline
Spinach
bird
reptile
calculus
ebstracce instraction
herb calculus
Paim
Canary crocodile
platypus
snake
Namander
turtle
Smmphibaan
lettuce
monetreme
chicken
Quality
mammal
equaiity
crysttal
mammall
pine
atbibbuee
monotrems
monksy
similarity
woody plant
frog
animal
parrot
amphibian
crystal
primate
object
cameness
animal
entity
living thing
ddeniity
eentity
spinacch
homo Sapiens
frog
plant
Calamander
tiger
platypus
hamo, sapiens
abject
oneness
lkknees
alligator
living thing
opprossmation
chicken
primate
feline
canary
alligaor
lettuce
simitarity
hater
cat
ddetiiy
natural objject
Parrst
crocodile
Samenes approhiman inrtimmtttion
@ monkey
oneness
woody plant
aftrihute
natural object
equality
turte ennake
ikenees abbstr
Word2Vec Vec reconstruction using MCM
(b) Word2Vec reconstruction using TIM
Smillaity
lettuce
playyess
turtle
reptile
likeness
onemess
natural, object
herb equality quality
likeness
living thing
approximation
homa, sapiens
impribaan
calculus
primate
amphiliian attribute
mammal
crystal
monotreme
calculus
natural objer poject
feline
snaka
pine
mammal
palm
approdiman approcimation
monkey
similarity
Uger
plant
aameness
enttty
ahstr attractton
plant
entity
crocodile
canary
object
reptile
abstraction
bird
spinach
Palm
crocodile
quality
canary
Chicken
lettuce
alligator
Stetity
parrt
sameeness
weedy plant inimal
alligetor
lieer
trog
living thing
salamandes dee
homo sapiens
Saay plant
teline teline pincc
crysta!
Salamander
dentity
attrbutt
equality
rock
monkey
turtle
bad
(c) Glo Ve-2BB reconstruction using MCM
(d) Glo GloVe-42B reconstruction using TIM
Figure 8: Knowledge graph reconstruction using Word2Vee and GloVe.
15

---

Under review as a conference paper at ICLR 2021
thing Canary Suurll
feline homo, sapiens
crocodile
homo sapiens
Huing, thing
living thing
chicken
monstreme
salamander
parros snake
monnka
frog
eimilaniy R
piant
natural abject
abstra
triphii
parrot
primate
Ballipatr crocodile
aeeniity
onemeess
approxxine psproximation
herli spinach
Caat feline
repiile
quality
Samehess
orenesg
approsimation
qualitt
entity
animal
mamman hwd
palm
pine
alligator
platypus
canary
samenies arophiplan
antmnal
Centity
atstr serractionn
likeness
object
attribbute
manihal
natural, abject rock
equality
tiger woody plunt
primate
palo hntontrrmme
identity
amblarit
saiam saiamandic Ay/Ti
likeness
rock objecc
calculus
crystal
montayy
Balatauu
reptile
shaki
tiger
chicken
pine
herb
bird
spinach
woody plant
inttuce plant
(a) GPT-2-XI. reconstruction rec----...ccoommuuttiini
using MCM
(b) GPT-2-XI. reconstruction using TIM
Salamander
homa, sapiens
monkey
lettce
alligstor
Parrot
tiyer
canary
crocadile
similarity turtis
Conetate plan
mamina
parrot
rock
crocodile
planiamert aamenee
felline
crystai
approximation
snake
plant
herb
sameeess
billgntor
animal
livingg thing
trog
canary
@ turtle
Smake
woodly piiant bird
living thing
reptile
rock
oneness
calculus
natural doject
car
reptile
chicken
herD
entlity
platypus
chicken
trog
monotreme
emtiyy
mammaal
@ pine
spinach
monetrema ereeeemm
spinach
similarity
hatural object mankey
amppotiinn
oneniess
animal
platyous
crystal
paim calculus salimaation
feline
tiger
amphibian palm mailty menkky uailiyy
abjject
likéness
attribute
abstraction
object
primate
homo supiens
lettuce
attrbuute
abstra ctttn
spprocmation
quality
equality
identry
(c) large reconstruction using MCM
(d) Ta-lerge reconstruction using TIM
Figure 9: Knowledge graph reconstruction using GPT-2 and T5.
16

---

spinach Under review as a conference paper at ICLR 2021
h Bhidkan
canary
spinach
parrot tiger Chome taplens
monkay snalle
primate
amphittian
monetremme
Aalm
wendy frocdilee
lettuce
plant
Shidd car
calculuu
living thing
naturall object
Dobject
animal
pine
amppiibian
snake
monkey
alligaoy
crocortile
turtie
feline
reptle
natural, objject
entivyl animal
mammal
platypus
spinach
satamannd dadeer
approximation
lilenesy
simiianl attritutts
entitty
manomal
feline
alligutor
rock
eryttal
Miving thing
plant plant
woody
palm
identity
abitaction
quality
Niger parril grmaie puss
masmanarenn pla
frog
oneness
approicmation
abstinct anstraction
Quality
object attribute
pine @ herb
canary
equaiity
oniiess
tarmeness
reptile daamander turtie
homa Sapiens
bird
tikenes
equaliy
Ideenity
esmmlaritt E Eleullaecce
sameness
(a) large reconstruction using MCM
(b) ELrge reconstruction using TIM
Jurtle
palm pine
herb
spinach
plant
chickeen
woedy plant
lettuce
bird tiper
parrot
pine
turtle
primmte
wondy piant
plant
herb
reptiie
manatreme
letture
emphibaan
abject
ebstraction
Frecodil galm
abjject
Whstraction
approximation
living thing
shake
livingg thing
heno sapiens
feline
animaal
natural, onject
eentity
platyRus
smake
mummmal
krimael
netural oblecc
Sommmann likeness
prisaae
attribute
simiarity
onemess
parrot
felin
atrriuua
quality
Cartennte
alligator
qualit
sameness
ooveness ap roproummation
crocodile
cat.
mankay
denetyy
tiger
ca mander
ctysta aamping eapiens Pchicka Miche
platypua
salemander
spinach
calculius
crystal
equaiity
ailigtoo
monkey bird
rentlle
calculls rack
freg
(c) llage reconstruction using MCM
(d) BERT- -laree reconstruction using TIM
Caleudius crrstal
tock
nnsuurll object
quain similarity
samenits
abstraction
liteness
pareot
frog
platyous
herb
wondy plant
palm
lettore
Canary
Spiaah
object
snake
chicken
makke
niant
utigutar
menetreme
monotremee
primate
animal living ming
abstraction
pampt
enitity
attribuuee
marmi ampnitlan
hamoo Shieess
mantkyy
homo sapiéns
bur
entily naturall pteject
artribuue
quality
tige
crocudie
living thing
simtlarity
equality
ping wtody plant
turble
cat
feline
Ideatity
onieeets Bar
palm
animall
frog
herti
likente
canary
lettuce
repriie
Uger
reptile
mammaal
happro happroormmmmmm
plant
Ppnmare
platyruc cat
cale daleudus
sta
(c) RoBERTa-large reconstruction using MCM
(f) RoBERTa-large reconstruction using TIM
Figure 10: Knowledge graph reconstruction using ELMo, BERT and RoBERTa.
17

---

Under review as a conference paper at ICLR 2021
homme rsspeers raspeers
woody, plant
manomeme
onareess Quility
equuliity
approximation
lettuce
mammal
snaka
primare
identilyy
sameness eommarity
palm Swinach
rack
@ calcutuo
attribula
illnnes
herd
plant
crystal
abstraattion
marman marnnnal plant
Peddydy paant
herth
otjectt
teptlle
Naturral objler
stmilariyy by BBeeeess
calculus
chiction
animal
reptile
natural object
livingg thing
crystal
turtle
crocodaa
centity
attrilite
Mberiit
cammeres
entimal
enttty
oognees
leftuce
monstreme
cat.
platypus
bid
approki inprrniiaation
equality
chicken
turtle
onplie
birrd
fetine
pine
teper
manikey
Higator
datamander
michiiiion
Iying thing
nalm
crocodile
spinas frog
Salamanderr
mardley
hamo Saplens
tipa. parmot
telirie
canary
platypics
canary
(a) XLNet-large reconstruction using MCM
(b) XLNet-large reconstruction using TIM
equality
onject
atigelor
lethure chicken
manott
feline
canary Trociille
likeness
rpporacimation
ho Appneay
Saplent
reptile
equality aireness
Jikeness
identity
snake
p Pam
parrt
primate
omammal
Idertity
samenese
Similarity
spinach
spinach
herb
aniinal
abstrant
Indraction
Quality
approximation
sameness
altribuue
quality
sbbttricc sistrsiction
entity
natuuall Lobiedt
animmll brd monkey
turtle
pine
hamo sapiens
pine
eliigarr
woody plant
eeitity
naturall object
cimitarity rity
living thing
weody plant
canary
objecct
oit aitrrbuuee
crystal
tiper
crocoaile
cut Calcunis
Laizolus
mammal
reptoe salamander
Helline
frog
pain bind
frop
furtie
chickan
crystal
plant
letuce
bwing thingg
hert
niant
Jamphibian
monementt reee piatyauggget
(c) -.BRR--large reconstruction using MCM
(d) ALBERT--arge reconstruction using TIM
Figure 11: Knowledge graph reconstruction using XLNet and ALBERT
18

---

Under review as a conference paper at ICLR 2021
C
FURTHER ANALYSIS OF FACTORS' IMPACT IN MODELS PERFORMANCE
Medann F1 11-ccore
0.90
Wors2Vectr
Gioli--425
UPT22XL
0.75
MoBERTalarge
Ket-large
A: ALBERT -aslange
Concept. Depth the Wordfet Graph)
Figure 12: Relative depth in the WordNet graph: For each synset, we compared FI with depth score (0 for the root and 100 % for leaves) measuring differences between higher/lower level concepts. As seen here, NCE-based methods are significantly impacted by this variable.
Median an F1-score
6.6
Wond2Vector
T5-large
ELMo large
DERT-large
RoBERTa large
Net-large
WordNet Distance Between the Concepts
Figure 13: Graph distance between concepts: We measured the impact of the number of "hops" that separate two tested concepts on pair-wise Fl score. This chart reveals a strong correla- tion of all the models in this aspect. As an example of this phenomenon, closer relations such as (chihuahua, dog) are, in general, considerably casier to capture than distant relations such as (chihuahua, entity). For details on how we implement the distance in WordNet, check Appendix A.2
Median F1
Word2Vester
0.50
T5-large
ELMo-lenge
BERTiaargs
ResERt a-lange
13
110 1000
10007
100010
Concept Frequency llog Scale)
Figure 14: Concept frequency: In this Figure we evaluate if frequent concepts easier or harder to capture for these models. The frequency was computed by counting occurrences in the 38 GB of OpenWebText Corpus (http: / Skylion007 github. is http//enneeTeetoorpppeo chart shows that uncommon and highly frequent concepts are harder to be modeled. In particu- lar, NCE-based models and T5 are more sensitive to this factor.
The
19

---

Under review as a conference paper at ICLR 2021
0.30
6.00
6.85
Median F1 71-score
Median F1
0:00
Word2Vector
Gleve-228
GPT2 XL
GPT2
0.70
15-large
15-large
ELMo large
ELMo lange
BERT-Awge
BERT-Aagg
RoBERTo
ALBERT-xotarge
-odaage
12
10 Number of Direct or Indirect Child Nodes
25
25
Number of Direct or Indirect Parent Nodes
Figure 15: Number of parent and child nodes: We studied the number of parents or child nodes have an impact on F1 -coress.. The same phenomenons show up when we analyze only direct parents or children.
8.86
0.35
0.58
0.54
6.82-
0.32
0.00
Median F1-xcoe
0.30
8.70
WordZVeeter SlaVe 428 GPT2 XL
0.70
WordPVector
Median
BoVe
8.70
0.76
GPT2 XL
0.08
El
Ts-lagge
0.24
EiMo-large
15-lärge
6.72
6.72
0.70
0.79
ange
ALBERT volonge
ALBERT volorge
0.55
14
0.50
Number Senses
Sense Ranking
Figure 16: Number of Senses and Sense Ranking: We studied models are impacted by multi- sense concepts such "period", and by their sense ranking (how frequent Surprisingly contextualized models, and specially CE models have no significant impact by this factor, suggesting that these models are very effective deducing the correct context. These charts also suggest that these models may be considering context even more than the words themselves. This is intuitive for Masked-Langen Lanoo--aannggddeeeee such as BERT, but for others, such as GPT-2. Non-contextualized models are impacted by this factor, as expected.
those senses are).
based on their
0.50
0.50
Median Fi-ccore
0.40
0:40
WordZVester Girve-22
WordEvester
Glov-122
GFT2
TS-lagg
large
ELMo
BERT-Wege
BERT
RuSiENt walange
Rodtii udlange
0.72
XLNett ange
6.72
XLNvet longe
ALBERT aodorge
ALBERT Jolarge
Sibling
Number Synonyms
Figure 17: Number of sibling nodes and number of synonyms: We discover that these factors have very low impact on Fl-scores. The fact that siblings are not confused leads to similar conclusions as the ones provided in Figure T6
20

---

Under review as a conference paper at ICLR 2021
D ACCURACY COMPARISON
In Table we analyzed F1 scores for each model and best layer. Here we provide an analogous table
but reporting accuracies instead of Fl scores.
Emb. Size
Best Layer
Accuracy All Layers
Accuracy Best Layer
Family
Model
Al//Best Layer
NCE
Word2Vee
Mikolov al.
2013
300/-
.7427 .0124
GloVe-42B PPeninntonn et
2014
300
.7617 .0088
GPT-2 (Radford
2018
9984 / 768
6
.7858
.7756
0097
T5-small (Raffel al
2019
7168 512
.8075 = .0088
.8122
0062
GLM
GPT22xI (Radford
2018
78400 / 1600
13
.7910
.7911
.1022
T5-lange (Raffll et 2019
2019
51200 1024
17
.7988 = .0133
.8244
.0189
ELMo-smal (Peters et
20186
768 / 256
2
.7872 = .0106
.7754
,0106
BERT-base Deviin al.
2019
9984 / 768
10
.8234 .0100
.8062
.0093
RoBERTa--aase (1.iu
2019
9984 768
5
.8349 = .0033
.8120
.0083
XLNet-base Yang al.
2019
9984 / 768
4
.8267 .0092
.8176
.0106
CE
ALBERT-base (Lan ot
2020
9984 768
12
.8145 = .0143
.7970 = .0089
ELMo-large Peters
2018b
3072 / 1024
2
.8242 .0766
.8245
.0075
BERT-large Devlin al.
2019
25600 1024
14
.8221 = .0155
.8055
.0104
LoBERTa-large Liu
25600 / 1024
13
.8258
.8189
J0167
XLNet-large Yang et al.
2019)
25600 1024
.8215
.0105
.8119 .0099
ALBERT-xxlarge (1.an
53248 4096
4
.8215
.8550
Table 5: Accuracies for every model in this study. This table is analogous to Table
E PRE-TRAINING CORPUS COMPARISON
Pre- Pre-Training Corpus Size
Family
Model
Tokens
Uncompressed Size
Word2Vec
33B
150GB*
NCE
GloVe-42B
42B
175GB*
GPT-2
10B*
40GB
GLM
T5
180B*
750GB
ELMo
0.8B
4GB*
BERT
3.9B
16GB
CE
RoBERTa
38.7B*
160GB
XLNet
32.9B
140GB*
ALBERT
3.9B
16GB
Table 6: Pre-Training corpus sizes used for each one of the studied models. The official sources report corpus sizes in terms of number of tokens or uncompressed size in GB. The symbol denotes values estimated by us based on official available information.
21

---

Under review as a conference paper at ICLR 2021
F
ARE ALL THESE MODELS CORRELATED?
Campured RodeF1oan
WortPeeetr pave
To
Ri arge
BORT arge
BudERTe etge
BonTrarge
64
Xchoikarge
0.53
8.50
gloe Fliecore
robetta, wrge's Fl-scton
Figure 18: Analysis of Fc-score correlations among models. A horizontal line in these charts would represent perfect independence.
Verd2Vec
Glo
GPT-2
ELMo
BERT
RoBERTa
XLNet
ALBERT
Verd2Vec
.6743
.1388
.4568
.2453
.1713
.1582
.2899
.276
GloVe
.6743
.3885
.5071
.2255
.2511
.2133
2896
.3192
GPT-2
.1188
.3185
.4675
.6402
.7351
.7611
.7212
.7653
T5
.4568
.5071
.6755
.4826
.3991
.4122
.4704
.4597
ELMo
.2453
.925
.6402
.4826
.6446
.6634
.6674
.6279
BERT
.1713
.2511
.7351
.3991
.6146
.8096
.7655
.7916
RoBERTa
.1582
.2413
.7611
.4122
.6634
.8096
.7844
.7890
XLNet
.2189
.2896
.7212
.4704
.6674
.7655
.7844
.7513
ALBERT .2276
.3192
.7653
.4597
.6279
.7916
.7890
.7513
Table Pearson correlation among concept-level score obtained by different architectures. We only considered concepts with F1-scores between 0.55 an 0.9 remove possible noisy outliers.
In this section, we study concepts that are easy/hard to associate one model are also easy/hard to associate for other models. We do this by analyzing the correlation between outputs using the semantic probing classifier. Charts in Figure shww the correlation among the models by plotting the Fl-score of one model against the rest. We picked one model of each family: GloVe 18-1et) RoBERTa-large (Fig midddee,,, and Ta-large (Fig 18 right).
Figure left reveals strong semantic correlation between Word2Vec and GloVe, but weaker with the rest. Figure middle shows strong correlation between RoBERTa and other methods in its family, but weaker with models of the NCE family. Fig 18ght shows that T5 has the same diffi- culties as the rest of the methods, even showing some correlation with the NCE family of methods.
The charts in Figure 18 reveal that models are strongly correlated, specially within the same model family, suggesting that similar approaches lead to similar biases, regardless their size. other words, the concepts that were difficult in the past are, to great extent, still difficult for state-of-the-art models. Thus, novel inductive biases are needed to get closer to human semantic capabilities.
22

---

Under review as a conference paper at ICLR 2021
G ADDITIONAL FI-SCORES OF SEVERAL SEMANTIC CATEGORIES
Category
W2V
GloVe
GPT-2
T5
ELMo
BERT RoBERTa
XLNet ALBERT
.7142
.7296
.7224
.7662
.7088 1. E..200
.7118 Σ 1585
.7759 1.1837
.7712 - 1.1404
.6335 1 1.1732
abstraction
$ 2.1277
= 2.1194
#. 1.18
.992
.7044
.7213
.7730
.8033
.8094
.8677
.864
8050
attribute
2.1310
= .137
.9911
0863
+ .886
+ .0998
1.99
= .091
+ .0988
.6974
.7511
.7826
.7587
8049
8246
.2499
8066
8093
communication
.1330
.1244
.9667
.1083
.0125
- .0979
+ 00983
.1023
.7688 + .1320
.6929
.492
.7262
.6821
.6733
6256
.6491 - .1399
.858
group
= 11339
.2558
1179
+ .1711
- .2399
+ .2308
+ .2745
.7497
.7579
8155
.7668
.8312 .0707
8516 .U742
8552
.8422
8556
social group
.10088
.1466
.0833
U867
I .6988
.0724
.0819
.6200
.6648
.300
.6944
.6011
4804
4921
.5371
.277
taxonomic group
.1306
1300
20025
.1208
1583
- 2025
+ E199
_ 1944
+ .2305
7412
.7213
.3131
.6691
5461
5626
.379
.733
.437
family
$ .11363
= .244
= 2003
.1266
+ 1630
E .1377
E .1502
# .1626
$ .1705
6267
.6040
2567
.7156
.6167
3696
.201
.4555
2862
genus
.0989
= .1127
.1582
.1001
.1311
.1857
118555
.1853
.1945
7255
.7788
.7829
.7795
8163
8181
8229
.077
.208
psychological feature
1.12
.1016
.9904
.078
.0851
+ .0954
1 .0930
.0931
+ .0915
.7664
.7567
.7612
.7663
.6799
7662
.7649
7659
7727
relation
$ 5.1304
= .14022
.0099
.6888
.0788
1 .095
E .9822
.9908
.0129
.7120
.7389
.7903
.7868
.8388
8249
8315
.8319
8231
artifact
1.11
= .1068
.736
0676
= .6933
- .0742
+ .00000
= .0002
$ .0706
.7230
.7510
.7033
.7878
.8398
.8992
8363
.8393
.8322
covering
1 E.1097
= .0970
.0713
.606
i. .0599
1 .0706
i. .0571
# .0576
: .0621
.7644
.7378
.7300
.7022
8308
8134
8337
.8313
8233
instrumentality
- .12199
.1052
.0728
.6488
.676
+ .0748
+ .0691
.0711
+ .0763
.7097
7435
.7956
.7899
8311
.198
8358
.8326
8230
device
1. .1220
.1009
1 .0013
.6677
+. .0689
1 .0701
1 .0640
+ .0675
+ .0743
.7240
.7398
.7253
.7751
8022
7453
.7631
.7788
78266
causal agent
- 1111
= .11011
11005
00757
= .0884
= .1093
+ 111066
- 1035
.0334
.7208
.7351
.7207
.7735
.8022
.7379
.7585
.7779
.7826
person
1. 1.1135
1 .1113
.1132
.0746
1. .0854
i. .1101
1.1055
1. .1033
i .0937
.7295
.7421
.7300
.7862
.8887
.7593
.7823
.7907
.7758
living thing
+ 2.1112
= .1078
1004
.0:49
= .8500
+ .0997
4.0922
= .9288
.9009
.7349
.7391
.7515
.7837
.8389
.7914
.8179
.8135 .0737
.7781
animal
.1049
.1088
.0829
.0714
.0785
.0801
.7701
.0857
.7445
7608
.2888
8168
8404
.7679
79622
7986
7645
plant
1. 1.1044
.989
.0842
.0633
.6922
1. .0830
i .0688
1 .0746
.0861
matter
.7402
.7633
.7582
8002
7756
7645
7614
.636
7685
+ ..1117
.1009
.0834
.6622
.0886
+ .0959
+ 000911
.0908
.0338
part
.7532
.7759
.7540
8805
.7580
.499
.4411
.7526
.6100
+: .10000
.0852
.0947
substance
.7560
.7791
.7508 .0755
.0733
.7542 10828
.436
.7370
.477
.7500
.0967
.0809
.0567
.0952
+ .9199
.0662
.0889
Table 8: FI-score and standard deviation of several semantic categories evaluated on the larger version of each model. Each value was computed by considering the F1-score of all the concepts that belong to the analyzed category. This is not an extensive list and categories are somewhat imbalanced. Categories were selected based on the number of sub-categories they contained.
23


Reviews:
[{"review_id": "yARipHFQZzx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis work addresses the question about how pre-trained language models encode semantic information. It adapts the methodology proposed in Hewitt & Manning (2019) for syntax to semantics, using the WordNet structure instead of a syntactic structure of a sentence to encode distances among word representations. The paper analyzes how embedding models encode suitable information to recreate the structure of WordNet. The study also shows evidence about the limitations of current pre-trained language models, demonstrating that all of them have difficulties to encode specific concepts.\n \nQuality\n\nThe proposed idea is very interesting, but the paper does not give a complete picture of what probing tasks can show and what their limitations are. The contribution of the paper is not clear. What can we learn from the experiment of the paper? How can we improve current language models? How can we exploit the distilled information?\n\nMissing reference for semantic probing tasks\n\nYaghoobzadeh, Yadollah, et al. \"Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.\nPeters, Matthew, et al. \"Dissecting Contextual Word Embeddings: Architecture and Representation.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.\n\nMissing references for the usefulness of probing tasks\n\nSaphra, Naomi, and Adam Lopez. \"Understanding Learning Dynamics Of Language Models with SVCCA.\" Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.\n\nA recent one paper on the usefulness of probing tasks\n\nRavichander, Abhilasha, Yonatan Belinkov, and Eduard Hovy. \"Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?.\" arXiv preprint arXiv:2005.00719 (2020).\n \nClarity\n\n-       The paper is clear and well written even if some references about the soundness of probing tasks are missing (see above) and a related discussion is missing too. In fact, the results of probing tasks have been questioned (see the references above) because it is not clear if the use of supervision allows the representation to adapt to the task.\n-       The related work section is a list of contributions and successes in probing tasks without a clear narrative.\n-       WordNet is not introduced\n-       Many acronyms are not defined (e.g.: WSD, MLP)\n-       I think that this part is important and should be clarified (last paragraph of Section 3.2): “Tests based on linear transformations such as that proposed by Hewitt & Manning (2019) did not allow us to recover the WordNet structure, which indicates that the subspaces in which the word embeddings models encode the semantics are not linear”. Intuitions or even hypotheses about this behaviour are not given.\n \nOriginality\n\n-       The analysis includes recent models such as ALBERT and T5.\n-       The idea of using the WordNet taxonomy to adapt the model proposed in Hewitt & Manning (2019) is very interesting\n \nSignificance\n\nThe proposed idea is very interesting and also the methodology is sound, but the conclusions are weak:\n- It is intuitive that it is more difficult to encode distant relations than others.\n- The fact that models in the same family have similar results is not discussed\n- It is not explained why only the Princeton WordNet Gloss Corpus has been used and not larger datasets annotated with WordNet senses such as SemCor.\n- Usually the models are evaluated at each layer, here all the layers are concatenated making it more difficult to understand where semantic information is stored.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "s0uV3QiXG7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " The authors conduct a study investigating how different language models incorporate semantic information in their respective learned representations. Investigating language models on their performance in concept-level tasks is motivated by the importance of the ability to organize and understand concepts in human intelligence. Another motivation is that other studies on the semantics in language models are not conclusive according to the authors, especially in determining where the semantic knowledge lies within the language models. \nThe study is conducted by using a semantic probing classifier which – in short – is trained to determine whether two words (inputted as learned representations from the language models) are semantically related (according to WordNet) or not. This classifier also aids in recreating a sampled knowledge sub-graph from WordNet.\nThe experimental section contains the evaluation of the two tasks, firstly the classification as described above and secondly the KG reconstruction.\nThe main findings of the study can be summarized as follows:\n- The authors show experimentally that models coming from the same family are strongly correlated\n- The authors show the experimental outcomes of the tasks mentioned above\n- The authors show how the individual layers of the language models contribute to the underlying knowledge\n- The authors show for all models how they are affected by different semantic factors (9 different factors, including number of senses, graph depth etc.)\nThe paper is clearly written and understandable and includes enough details to understand the implementation of the semantic probing classifier. The appendix contains detailed outcomes of the different experiments which, together with the result section, give a good overview of the experimental results.\n\nMy recommendation is towards acceptance of the paper, because the authors contribute to a more detailed understanding on how language models incorporate semantic knowledge and where this knowledge might be located within the models. Exploiting those findings could potentially lead to an improvement on future models. Also, the findings per se give more insight on how the internals of large models process information, which is a step towards a more explainable AI.\n\nI have one question regarding the inter-comparability of the models; were the tested language models all trained on the same unlabeled textual data (or on data of comparable size), or did you use pre-trained models that were published alongside their respective papers? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "puMyYXKiQV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzed how well the previously proposed pre-training models could encode the underlying knowledge graph by defining probing classifiers. The probe classified is trained on top of the pre-trained contextual presentation models, such as non-contextualized word embeddings (e.g., Glove), contextualized word embeddings (e.g., BERT), and generative language models (e.g., GPT-2), and tried to reconstruct the structure of the knowledge graphs.\n\nThis paper is well-written. Readers will easily understand what this paper did and tried to reveal. While it was not sufficiently clear why this paper adopts this approach, I think the idea of the probing classifiers and knowledge graph construction is a reasonably good idea to reveal how well the pre-training models encode the underlying knowledge graph.\n\nThe cons of this paper are the lack of a profound analysis of the experimental results. In Section 5, this paper tried to reveal what knowledge the existing pre-trained models work well or not well by using some statistics (e.g., the relative depth). While I think this approach is also good, readers will need more detailed information about analysis results to inspire new ideas to improve semantic learning abilities. For example, the number of samples in each concept depth and wordnet distance between concepts changes. Therefore, it is better to estimate confidence intervals for readers to precisely understand how the differences of the median F1-scores is important (or not important) in each depth or distance. Second, it is better to analyze which semantic category the pre-trained methods work well or not well. I guess that the concept depths and frequencies hugely change depending on the concept categories. It is helpful if this paper also elucidates which semantic cateogy the existing method work well and not well. Thirdly, it is better if this paper shed light on how readers can improve the semantic learning abilities based on these results. Without these proposals, I think the contribution of this paper is limited.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0rFm6fgxSOq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper 1) introduces a method to use three types of text embedding methods (non-contextual, contextual, LM -based) to predict  word relatedness (as a binary classification problem) for pairs of words in wordnet 2) uses these relatedness scores to build proxies of the Wordnet graph 3) carry out experiments based on the two bullet points to compare semantic understanding abilities of the aforementioned embedding methods.\n\nMajor comments:\n* The paper carries out quite a few experiments with weakly connected goals. It looks like a combination of miscellaneous results based on a common (and limited) technique, rather than delivering a coherent message with interrelated takeaways from follow-up experiments. \n* The technique to probe models is quite restricted in that it is centered around single word concepts. Given that contextualized models are utilized, it seems like a rather handicapped investigation of very powerful models.\n* In section 4.3, authors try to make a point about correlations on visuals. This is a dangerous approach, and it would be much better to rely on numerical summaries of correlations. In fact, it is extremely hard to judge correlation by looking at pictures, because correlation needs to take into account the variability in an F1 metric with the other axis kept constant (only means are shown). A curve with less slope on Figure 3 might indicate a much higher degree of co-movement with the metric on the X axis if the randomness in y axis wrt at any point in x axis is very low. Authors should revisit statistical correlation, and preferably revamp this section. That said, I'm not quite convinced that it is a publication-worthy result to say similar methods (and each of the 3 buckets is very similar within) produce similar concept relatedness scores.\n* It's hard to understand how the proposed probing classifier is different than concatenating $M(x)$ and $M(y)$ and directly applying MLP on it. One can choose an MLP with custom first hidden layer size and activation function that would be functionally equivalent to what's being proposed.\n* There is definitely truth to the title, but I'd suggest not conflating the term \"knowledge graph\", which traditionally represents actual world knowledge, and not lexical databases.\n\nMinor comment\n* Please write something under section 3 (before 3.1). Given that it's not clear on a first pass that you're introducing two different methods in 3.2 and 3.3, the empty space is a good opportunity to tell the reader about this fact.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work addresses the problem of understanding how pre-trained language models are encoding semantic information, such as WordNet structure. This is evaluated by recreating the structure of WordNet from embeddings. The study also shows evidence about the limitations of current pre-trained language models, demonstrating that all of them have difficulties to encode specific concepts.

pros:
- good idea to reveal how well the pre-training models encode the underlying knowledge graph
- detailed understanding on how language models incorporate semantic knowledge and where this knowledge might be located within the models
- experiments show that models coming from the same family are strongly correlated
- the paper shows how individual layers of the language models contribute to the underlying knowledge
- analysis of the different semantic factors (9 different factors, including number of senses, graph depth etc.) 
- paper is clearly written and understandable and includes enough details to understand the implementation of the semantic probing classifier. 

cons:
- weakly connected goals, response from reviewers is string around 3 main topics, which is seen as many for a single scientific paper. It would be easier to focus only on one topic and make a clear conclusion,
- single word concepts while CE models are powerful in context,
- lack of a profound analysis of the experimental results
    - hard to understand which semantic category the pre-trained methods work well or not well,
    - clarification about the improvement of the semantic learning abilities based on these results.

Several of the identified issues have been answered in the author's rebuttal, however, the paper would still need more work to be accepted. Note also that the bar a this year ICLR conference is high and we encourage the authors to submit their updated work again at the next conference.
Decision: Reject