JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
Authors: Donghan Yu
Chenguang Zhu
Yiming Yang
Michael Zeng
Venue:  (2021)
Keywords: Pre-training
Knowledge Graph
Language Understanding
Graph Neural Network

Abstract: Knowledge graphs (KGs) contain rich information about world knowledge, entities, and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains a challenge to efficiently integrate information from KG into language modeling. And the understanding of a knowledge graph requires related context. We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and relations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experimental results on several knowledge-aware NLP tasks show that our proposed framework achieves superior performance by effectively leveraging knowledge in language understanding.

Under review as a conference paper at ICLR 2021
JAKET: JOINT PRE-TRAINING OF KNOWLEDGE GRAPH AND LANGUAGE UNDERSTANDING
Anonymous authors
Paper under double-blind review
ABSTRACT
Knowledge graphs (KGs) contain rich information about world knowledge, tities, and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains challenge to efficiently integrate infor- mation from KG into language modeling, And the understanding of a knowledge graph requires related context. We propose novel joint pre-training framework. JAKET, to model both the knowledge graph and language. The knowledge mod- ule and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and re- lations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experimental results on several knowledge-awan NLP tasks show that proposed framework achieves superior performance by effectively leveraging knowledge in language understanding.
INTRODUCTION
Pre-trained language models (PLM) leverage large-scale unlabeled corpora conduct self- supervised training. They have achieved remarkable performance in various NLP tasks, exemplified by BERT (Devlin al., 2018). RoBERTa (Liu et al. 2019b), XLNet (Yang al. 2019 J and GPT series (Radford et al. 2018 2019 Brown et al. 2020 It has been shown that PLMs can effectively characterize linguistic patterns in text and generate high-quality context-aware representations (Liu
et al. 2019a). However, these models struggle to grasp world knowledge about entities and rela- Poerner al. 2019) Talmorr et al. 2019), which are very important in language understanding.
tions
Knowledge graphs (KGs) represent entities and relations in a structural way. They can also solve the sparsity problem in text modeling. For instance, a language model may require tens instances of the phrase "labrador is kind of dog" in its training corpus before it implicitly learns this fact. In comparison, a knowledge graph can use two entity nodes "labrador", "dog" and a relation edge "is a" between these nodes precisely represent this fact.
Recently, some efforts have been made to integrate knowledge graphs into PLM. Most of them combine the token representations in PLM with representations of aligned KG entities. The entity embeddings in those methods are either pre-computed based on an external source by a separate model (Zhang et al., 2019 Peters et al. 2019), which may be easily aligned with the language representation space, or directly learned as model parameters (Févry et al. 2020 Verga et al. 2020). which often have an werr-parameterization issue due to the large number of entities. Moreover, all the previous works share common challenge: when the pre-trained model is fine-tuned in new domain with a previously unseen knowledge graph, struggles to adapt to the new entities, relations and structure.
Therefore, we propose JAKET, a Joint pre-trAining framework for KnowledgE graph and Text. Our framework contains a knowledge module and language module, which mutually assist each other by providing required information to achieve more effective semantic analysis. The knowl- edge module leverages a graph attention network (Veličković et al.) 2017) to provide kructure-awar entity embeddings for language modeling. And the language module produces contextual represen- tations initial embeddings for KG entities and relations given their descriptive text. Thus, in both modules, content understanding is based on related knowledge and rich context. On one hand, the joint pre-training effectively projects entities/nelations and text into a shared semantic latent space,

---

Under review as a conference paper at ICLR 2021
Context Information
Pretraining:
KG
Language Module
Knowledge Module
Fine- Fine-tuning:
KG
Knowledge Information
Figure 1: simple illustration on the novelty of our proposed model JAKET.
which eases the semantic matching between them. On the other hand, the knowledge module produces representations from descriptive text, it solves the over-parameterization issue since entity longer part of the model's parameters.
embeddings
In order to solve the cyclic dependency between the two modules, we propose a novel two-step language module LM, and LM2, respectively. LM, provides embeddings for both LM2 and KG. The entity embeddings from KG are also fed into LM2, which produces the final representation. LM1 and LM2 can easily established as the first several transformer layers and the rest layers of pre-trained language model such as BERT and RoBERTa. Furthermore, we design an entity context embedding memory with periodic update which speeds up the pre-training by 15x.
The pre-training tasks are all sulf--uper, including entity category classification and relation type prediction for the knowledge module, and masked token prediction and masked entity predic- tion for the language module.
great benefit of our framework is that can easily adapt to unseen knowledge graphs the fine- tuning phase. As the initial embeddings of entities and relations come from their descriptive text, JAKET is not confined to any fixed KG. With the learned ability to integrate structural information during pre-training, the framework is extensible to novel knowledge graphs with previously unseen entities and relations, as illustrated in Figure
We conduct empirical studies several knowledge-awan natural language understanding (NLU) tasks, including few-shot relation classification, question answering and entity classification. The results show that JAKET achieves the best performance compared with strong baseline methods on all the tasks, including those with a previously unseen knowledge graph.
RELATED WORK
Pre-trained language models have been shown to be very effective in various NLP tasks, including EL.Mo (Peters al. 2018 GPT (Radford et al. 2018), BERT (Devlin et al.) 2018), RoBERTa (Liu ct al. 2019b) and XLNet Yang et al. 2019). Built upon large-scale corpora, these pretrained models learn effective representations for various semantic structures and linguistic relationships. They are trained on self-supervised tasks like masked language modeling and next sentence prediction.
Recently, a lot of efforts have been made on investigating how to integrate knowledge into
PLMs (Levine et al., 20199] Soares et al. 2019 Liu et al. 2020 Guu et al. 2020). These approaches can be grouped into two categories:
1. Explicitly injecting entity representation into the language model, where the representations are either pre-computed from external sources (Zhang al. 2019 Peters et. 2019) or directly learned as model parameters (Févry et al.) 2020 Verga et al. (020). For example, ERNIE (THU) (Zhang ett al. 2019) pre-trains the entity embeddings on a knowledge graph using TransE (Bordes et al., 2013 while EAE (Févry et al.] 2020) learns the representation from pre-training objectives with all the other model parameters. K-BERT (Liu et al. 2020) represents the entities by the embed- dings of surface form tokens (i.e. entity names), which contains much less semantic information compared with description text. Moreover, it only injects KG during fine- tuning phase instead of prett-nge KG and text.
2

---

Under review as a conference paper at ICLR 2021
Output: Context Representation
Entity Representation
Pretrain
Pretrain
Language Model 2
Masked entity prediction:
Entiyy category prediction: 02: Earth C10:
02:
Graph Convolution Network
Masked taken prediction: [MA4S]
Information Fusion
Relation type prediction:
(02) Farth, Q544: System) P361: part
Entity Embedding Memory
Tied
Language Model
Language Model 1
CB225 Sun
support
Input:
part_of
is the most importent [MASK] of energy for life on Earth.
Text
Entity Description Text
KG
cotegory
Q544: Solar System
The Sun la the star at the center of Solar System..
Earch the third planet from the Sun...
Figure 2: A demonstration for the structure of JAKET, where the language module is on the left side marked green while the knowledge module is on the right side marked blue. Symbol X indicates the steps to compute context representations introduced in Section 3.4 "Qx", "Px" and "Cx" are the indices for entities, relations and categories in KG respectively. Entity mentions in text are underlined and italicized such as Sun.
2. Implicitly modeling knowledge information, including entity-level masked language model- ing (Sun et al. 2209bb Shen et al., 2020), entity-based replacement prediction (Xiong et al. 2019) and knowledge embedding loss as regularization (Wang et al. 2019b). For example, besides token- level masked language modeling, ERNIE (Baidu) (Sun al. 2019b) uses phrase-level and entity- level masking to predict all the masked slots. KEPLER (Wang et al. 2019b) calculates entity em- beddings using pre-trained language model based on the description text, which is similar to our work. However, they use the entity embeddings for the knowledge graph completion task instead of injecting them into the language model.
Some works (Ding et al. 2019 Lv et al. 2020) investigated the combination of GNN and PLM. For example, Lv al. (2020) uses XLNet to generate initial node representation based on node context and feeds them into a GNN. However, these approaches do not integrate knowledge into language modeling, and they are designed for specific NLP tasks such as reading comprehension or common- sense reasoning. In comparison, we jointly pre-train both the knowledge graph representation and language modeling and target for general knowledge-aware NLU tasks.
METHOD
In this section, we introduce the JAKET framework of joint pre-training knowledge graph and lan- guage understanding. We begin by defining the mathematical notations, and then present our model architecture with the knowledge module and language module. Finally, we introduce how to pre- train model and fine-tune it for downstream tasks. The framework is illustrated in Figure 2
3.1 DEFINITION
knowledge graph denoted by KGG (E, R, T), where E {er ... ..t....e] is the set of entities and
R = -fr . is the set of relations. T = ri, [e)]e < i < T, ee E E, e R} stands for the set of head-relation-taal triplets. N, ((,,,)((,,r, E T) represents the set of neighboring relations and entities of an entity v.
vocabulary of tokens and the contextual text x [T1 x2. T as a sequence of tokens where x; E V. In the vocabulary, [MASK] is the special token for masked language modeling Devllnn et al. 2018) and [CLS). [EOS] are the special tokens indicating the beginning and end of the sequence. We define F as the dimension of token embeddings, which is equal to the dimension of entity/relation embeddings from the KG.
We define V [MASK], [CLS], [EOS], W1 ..... wy
The text x has a list of entity mentions m = [m]...., m.l. where each mention m, (Emgg Smay Omg ): Cip is the corresponding entity and sm,; Om, the start and end index of this
3

---

Under review as a conference paper at ICLR 2021
mention in the context. In other words, [Zsm, ...... Zom, is linked with entity ema span mentions are disjoint for a given text sequence.
We assume the
As entities in the knowledge graph are represented by nodes without context, tion text to describe the concept and meaning of entities. For each entity 6₁ its description text x deseribes this entity. The mention of 6; in x is denoted as m (e;, s.. of similarly defined as above. For instance, the description text for the entity "sun" can be "[CLS] The Sun is the star the center of the Solar System [EOS]". Then the mention m (Sun, 3, 3). If there are multiple mentions of e; in its description text, we choose the first one. If there's s no mention of e; its description text, we set sf of = Similarly, we define relation description text the text that can describe each relation.
entity descrip-
3.2 KNOWLEDGE MODULE
The goal of the knowledge module (KM) is to model the knowledge graph to generate knowledge- based entity representations.
To compute entity node embeddings, we employ the graph attention network (GAT) (Veličković 2017). which uses the self-attention mechanism specify different weights for different neighboring nodes. However, the vanilla GAT is designed for homogeneous graphs with single- relation edges. To leverage the multi-relational information, we adopt the idea of composition op- erator (Vashishth al. 2019) to compose entity embeddings and relation embeddings. In detail, in the 1-th layer of LM, we update the embedding Ee()) of entity as follows:
E(D
LayerNorm
Σ (naa)NN
oe W f(E Ro)) + EE!-1)
km
exp LeakyReLL
(aT [W* w S(E"'" R.))))
0,, Eer exp
LeakyReLU (aT
W*E W* ((E Rr))))
where LayerNorm stands for layer normalization (Ba et al. 2016 + means concatenation and K the number attention heads. W the model parameter and R, is the embedding of relation Note that the relation embeddings are shared across different layers. The function f(., R x RF - R merges a pair of entity and relation embeddings into one representation. Here, we set y inspired by TransE (Bordes al. 2013). More complicated functions like MLP
f(x
network can also applied.
The initial entity embeddings (()) and relation embeddings R are generated from our language module, which will be introduced in Section 3.4 Then, the output entity embeddings from the last GAT layer are used as the final entity representations EKM Note that the knowledge graph can be very large, making the embedding update over all the entities in Equation we follow the minibatch setting (Hamilton et al., (017)): given a set of input entities, we perform neighborhood sampling generate their multi-hop neighbor sets and we compute representations only on the entities and relations that are necessary for the embedding update.
(I) not tractable. Thus
3.3 LANGUAGE MODULE
The goal of the language module (LM) model text data and learn context-aware representations. The language module can be any model for language understanding, e.g. BERT (Devlin et al.) 2018 In this work, we use the pre-trained model RoBERTa-base (Liuu et al. 2019b) as the language module.
3.4 SOLVING THE CYCLIC DEPENDENCY
In our framework, the knowledge and language modules mutually benefit each other: the language module LM outputs context-aware embedding initialize the embeddings of entities and relations in the knowledge graph given the description text; the knowledge module (KM) outputs knowledge- based entity embeddings for the language module.
We do consider discontinous entity mentions in this work.
4

---

Under review as a conference paper at ICLR 2021
However, there exists a cyclic dependency which prevents computation and optimization in this design. To solve this problem. we propose a decomposed language module which includes two lan- guage models: LM, and LM2. We employ the first 6 layers of RoBERTa as LM, and the remaining 6 layers as LM2. The computation proceeds as follows:
1. LM1 operates on the input text x and generates contextual embeddings Z.
2. LM, generates initial entity and relation embeddings for KM given description text.
3. KM produces its output entity embeddings to be combined with Z and sent into LM2-
4. LM2 produces the final embeddings of x, which includes both contextual and knowledge information.
In detail, in step suppose the context x embedded as Xeembed LM1 takes X Xembed as input and outputs hidden representations Z = LM, (Xembed).
In step suppose x is the entity description text for entity Ej, and the corresponding mention is m = (eg, sj. of). LM, takes the embedding of x and produces the contextual embedding z Then, the average of embeddings at position sj and of used the initial entity embedding of ey,
i... E [ + Z /2. The knowledge graph relation embeddings R are generated in a similar way using its description text.
In step 3, KM computes the final entity embeddings ΕKМ which then combined with the output Z from LM, In detail, suppose the mentions in x are m positions of mentions:
Z and EKM are combined
Zx. +
AKM Emm
if 3i s.t. 8m; S k S Om,
merge
(3)
k
Zk
otherwise
where KK
is the output embedding of entity cm, from KM. Then we apply layer normalization (Ba mergge Z' LLyerNormm (Zmerorrmmmmeg)) merggg)) Finally, Z' is fed into LM2.
cm
et al.
2016
In step 4, LM2 operates on the input Z' and obtains the final embeddings ZLM LM2(Z'). The four steps are marked by the symbol { in Figure 2 for better illustration.
3.5 ENTITY CONTEXT EMBEDDING MEMORY
Many knowledge graphs contain large number of entities. Thus, even for one sentence, the number of entities plus their multi-ho hop neighbors can grow exponentially with the number of layers in the graph neural network. As a result, it's very time-consuming for the language module to compute context embeddings based on the description text of all involved entities in a batch on the fly.
To solve this problem, we construct an entity context embedding memory, Econtent, to store the initial embeddings of all KG entities. Firstly, the language module pre-computes the context em- beddings for all entities and places them into the memory. The knowledge module only needs to retrieve required embeddings from the memory instead computing them, i.c. E(0) + geontent
However, embeddings in the memory computed from the "old" (initial) language module while the token embeddings during training are computed from the updated language module, there will be undesired diserepancy. Thus, we propose update the whole embedding mem- ory goontezt with the current language module every T ((i) steps, where i is the number of times that the memory has been updated (starting from 0). T(i) set as follows:
T(1) min(Iina s/r] s/r1,Imaz) Imaz)
(4)
where Linit is the initial number of steps before the first update and a is the increasing ratio of updating intervals. is the number of repeated times of the current updating interval. Imaz is the maximum number of steps between updates. [-] means the operation of rounding down. our experiments,
Innt 10, 2,r 3. Imaz 500, and the corresponding sequence of T [10, 10, 10, 20, 20, 20, 40, 40, 40. .. 500, 500]. Note that we choose > because the model parameters usually change less training proceeds.
5

---

Under review as a conference paper at ICLR 2021
Moreover, propose momentum update to make Exontent evolve smoothly. Suppose the newly calculated embedding memory by LM is Enew then the updating rule is:
grententt b gountest + m m.tnnttst + (1 m) Bnew
(5)
where m E [0, 1) is a momentum coefficient which is set 0.8 in experiment.
This memory design speeds up our model by about 15x during pre-training while keeping the effec- tiveness of entity context embeddings. For consideration of efficiency, we use relation embeddings only during fine-tuning.
3.6 PRE-TRAINING
During pre-training, both the knowledge module and language module are optimized based on sev- eral self--supr learning tasks listed below. The examples of all the training tasks are shown in Figure
At each pre-training step. first sample batch of root entities and perform random- aandom-walk walk sampling on each root entity. The sampled entities are fed into KM for the following two tasks.
Entity category prediction. The knowledge module is trained to predict the category label of entities based on the output entity embeddings FKM, The loss function is cross-entropy for multi- class classification, denoted as Le.
Relation type prediction. KM is also trained to predict the relation type between a given entity pair based on ERM The loss function is cross-entropy for multi-class classification, denoted Cy.
Then, we uniformly sample batch of text sequences and their entities for the following two tasks.
Masked token prediction. Similar to BERT, We randomly mask tokens in the sequence and predict the original tokens based on the output ZLM of the language module. We denote the loss as L.-
Masked entity prediction. The language module is also trained to predict the corresponding entity of a given mention. For the input text, we randomly remove 15% of the mentions m. Then for each removed mention my (er, Sr, 0p), the model predicts the masked entity e, based the mention's ' embedding. In detail, it predicts the entity whose embedding in geontezt is closest to q == = )/2), where g(x) GELU(>W; WW2 a transformation function. GELU an activation function proposed by Hendryckk & Gimpel (2016). Since the number of entities can be very large, we use e, neighbours and other randomly sampled entities as negative samples. The loss function C. is cross entropy based on the inner product between and each candidate entity's 's embedding. Figure 2) shows an concrete example, where the mention "Earth" is not marked in the input text since it masked and the task is to link the mention "Earth" to entity "Q2: Earth".
3.7 FINE-TUNING
During fine-tuning, our model supports using either the knowledge graph employed during pre- training or novel custom knowledge graph with previously unseen entitie If custom KG is used, the entity context embedding memory is recomputed by the pre-trained language module using the new entity description text. In this work, we do not update the entity context memory during fine-tuning for consideration of efficiency. We also compute the relation context embedding memory using the pre-trained language model.
EXPERIMENT
4.1 BASIC SETTINGS
Data for Pre-training. We use the English Wikipedia the text corpus, Wikidata (Vrandečić & Krötzsch 2014) as the knowledge graph, and SLING (Ringgaard et al. (017)) to identify entity men- tions, For each entity, we use the first 64 consecutive tokens of its Wikipedia page as its description text and we filter out entities without a corresponding Wikipedia page. We also remove entities that
We assume the custom domain comes with NER and entity linking tools which can annotate entity men- tions in text. The training of these systems heyond the scope of this work.

---

Under review as a conference paper at ICLR 2021
Model
5-way I-shot
5-way 5-shot
10-way 1-shot
PAIR (BERT)"
85.7
89.5
76.8
PAIR (RoBERTa)
86.4
90.3
77.3
PAIR (ROBERT++GNN)
86.3
PAIR (ROBERTA+GNN+NN)
86.9
PAIR (KnowBERT)
86.2
90.3
77.0
PAIR (JAKET)
87.4
92.1
78.9
Table 1: Accuracy results on the dev set of FewRel 1.0. * indicates the results are taken from Gao al. 2019). PAIR is the framework proposed by Gao et al. (2019).
have fewer than 5 neighbors in the Wikidata KG and fewer than 5 mentions in the Wikipedia cor- pus. The final knowledge graph contains 3,657,65 7658 entities, 799 relations and 20, 10,113,978 triplets. We use the instance of relation to find the category of each entity. In total, 3,039,909 entities have category labels of 19,901 types. The text corpus contains about 4 billion tokens.
Implementation Details. We initialize the language module with the pre-trained RoBERTa- base (Liu et al. 2019b) model. The knowledge module is initialized randomly. Our implementation based on the HuggingFace framework (Wolf et al., 2019) and DGL (Wang al., 2019a). For the knowledge module, we use a 2-layer graph neural network, which aggregates 2-hop neighbors. The number of sampled neighbors in each hop is 10. More details are presented in the Appendix.
Baselines. We compare our proposed model JAKET with the pre-trained RoBERTa basse (Liu et al., 2019b) and two variants of our model: ROBERTa+GNN and RoBERTa+GNN+M. The two models have the same model structure JAKET, but they are not pre-trained on our data. Moreover, the entity and relation context embedding memories of RoBERTa+GNN are randomly generated while the memories of RoBERTa+GNN+M are computed by RoBERTa.
4.2 DOWNSTREAM TASKS
Few-shot Relation Classification. Relation classification requires the model to predict the rela- tion between two entities text. Few-shot relation classification takes the N-way K-shot setting. Relations in the test set are not seen in the training set. For each query instance, N relations with K supporting examples for each relation are given. The model is required to classify the instance into one of the N relations based on the N x K samples. In this paper evaluate our model on FewRel (Han et al. 2018), which a widely used benchmark dataset for few-shot relation classifi- cation, containing 100 relations and 70,000 instances.
We use the pre-trained knowledge graph for FewRel as it comes with entity mentions from Wikidata knowledge graph. To predict the relation label, we build sequence classification layer on top of the output of LM. More specifically, we use the PAIR framework proposed by [Gao et al. (2019), which pairs each query instance with all the supporting instances, concatenate each pair as one sequence, and send the concatenated sequence to our sequence classification model to get the score of the two instances expressing the same relation. We do not use relation embeddings in this task to avoid information leakage.
As shown in Table D our model achieves the best results in all three few-shot settings. Comparing the results between RoBERTa and ROBERTa+GNN, we see that adding GNN with randomly gener- ated entity features does not improve the performance. The difference between ROBERTa+GNN+M and RoBERTa+GNN demonstrates the importance of generating context embedding memory by the language module, while JAKET can further improve the performance by pre-training. We also compare with a strong Knowledge-enhanced PLM KnowBERT (Peters et al. 2019)). which is also pretrained on English Wikipedia and Wikidata KG. The results show that JAKET consistently out- perform KnowBERT in different few-shot settings.
KGQA. The Question Answering over KG (KGQA) task is to answer natural language questions related to a knowledge graph. The answer to each question is an entity in the KG. This task requires an understanding over the question and reasoning over multiple entities and relations.
We use the vanilla version of the MetaQA (Zhang et al. 2017) dataset, which contains questions requiring multi-hop reasoning over a novel dovie-domain knowledge graph. The KG contains 135k triplets, 43k entities and relations. Each question is provided with one entity mention and the
7

---

Under review as a conference paper at ICLR 2021
Model
KG-Full
KG-50%
Model
100%
20%
5%
1-hop
2-hop
I-hop
2-hop
GNN
48.2
RoBERTa
90.2
70.8
61.5
39.3
RoBERTa
33.4
RoB+G+M
91.4
72.6
62.5
40.8
RoB+G+M
79.1
66.7
53.5
JAKET
93.9
73.2
63.1
41.9
JAKET 81.6
70.6
58.4
Table 2: Results the MetaQA dataset over 1- hop and 2-hop questions under KG-Full and KG- 50% settings. RoB+G+M the abbreviation for the baseline model RoBERTa+GNN+M.
Table 3: Results on the entity classifica- tion task over unseen Wikidata knowledge graph. RoB+G+M is the abbreviation for the baseline model RoBERTa+GNN+M.
question is named as k-hopp question the answer entity is a k-hop neighbor of the question entity. We define all the k-hop neighbor entities of the question entity as the candidate entities for the question. We also consider more realistic setting where we simulate an incomplete KG by randomly dropping a triplet with a probability 50%. This setting called KG-50% compared with the full KG setting KG-Fulli For each entity, we randomly sample one question containing it as the entity's 's description context. We manually write the description for each relation since the number of relations is very small. We use the output embedding of [CLS] token from LM as the question embedding, and then find the entity with the closest context embedding.
As shown in Table 2 RoBERTa+GNNN+ outperforms RoBERTa, demonstrating the effectiveness of KM+LM structure. JAKET further improves the accuracy by 0.6% to 2.5% under both KG settings, showing the benefits of our proposed joint pre-training
Entity Classification. To further evaluate our model's capability to reason over unseen knowledge graphs, design entity classification task. Here, the model is given portion of the Wikidata knowledge graph unseen during pre-training, denoted as KGG'. It needs to predict the category labels of these novel entities. The entity context embeddings are obtained in the same way as in pre- training. The relation context embeddings are generated by its surface text. The number of entities and relations in the K.G' are 23,046 and 316 respectively. The number of triplets is 38,060. Among them, 16,529 entities have 1,291 distinct category labels. We conduct experiments under semi- supervised transductive setting by splitting the entities in KCG' into train/dev/test splits of 20%, 20% and 60%. To test the robustness of models to the size of training data, we evaluate models when using 20% and 5% of the original training set.
In this task, RoBERTa takes the entity description text as input for label prediction while neglecting the structure information of KG. JAKET and ROBERTa+GNN++M make predictions based on the entity representation output from the knowledge module. We also include GNN as a baseline, which uses the same GAT-based structure as our knowledge module, but with randomly initialized model parameters and context embedding memory. GNN then employs the final entity representations for entity category prediction.
As shown in Table 3 our model achieves the best performance under all the settings. The per- formance of GNN or RoBERTa alone significantly lower than JAKET and RoBERT=+GNN+M, which demonstrates the importance of integrating both context and knowledge information using our proposed framework. Also, the gap between JAKET and ROBERTTa+GNN+M increases when there's less training data, showing that the joint pre-training can reduce the model's dependence downstream training data.
5 CONCLUSION
This paper presents a novel framework, JAKET, to jointly pre-train models for knowledge graph and language understanding. Under our framework, the knowledge module and language module both provide essential information for each other. After pre-training, JAKET can quickly adapt unseen knowledge graphs new domains. Moreover, we design the entity context embedding memory which speeds up the pre-training by 15x. Experiments show that JAKET outperforms baseline methods in several knowledge-aware NLU tasks: few-shot relation classification, KGQA and entity classification. In the future, plan extend our framework natural language generation tasks.
For fair comparison, we do not include models which incorporate dedicated graph retrieval module (Sun 2018 2019a)
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 06450, 2016.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pp. 2787-2795, 2013.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension scale. arXiv preprint (tti::9905.55000 2019.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. En- tities as experts: Sparse memory access with entity supervision. arXiv preprint arXiv:2004.07202, 2020.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fewrel 2.0: Towards more challenging few-shot relation classification. arXiv preprint arXiv:1910.07124, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wer Wei Chang. Realm: Retrieval- augmented language model pre-training arXiv preprint 2kki::2002.08009 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs In Advances neural information processing systems, pp. 1024-1034, 2017.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. arXiv preprint arXiv: 1810. 10147, 2018.
Dan Hendrycks and Kevin Gimpel. 1rXivv1606.0844 2016.
Gaussian error
linear units (gelus).
arXiv preprint
Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwards Amnon Shashua, and Yoav Shoham. Sensebert: Driving some sense into bert. arXiv preprint arXiv: 05646, 2019.
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. Linguistic knowledge and transfcrability of contextual representations. arXiv preprint arXiv:1903.0883. 08855, 2019a.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph. In AAAI, pp. 2901-2908, 2020.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 11692, 2019b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang. Guihong Cao, and Songlin Hu. Graph-based reasoning over heterogencous external knowl- 2020.
edge for commonsense question answering. In AAAI, pp.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint wrXiv:1802.05365, 2018.

---

Under review as a conference paper at ICLR 2021
Matthew E. Peters, Mark Neumann, Robert L Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In EMNLP, 2019.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. Bert is not knowledge base (yet): Factual knowledge vs. name- based reasoning in unsupervised qa. arXiv preprint arXiv:1911.0368) 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8)9, 2019.
Michael Ringgaard, Rahul Gupta, and Fernando CN Pereira. Sling: A framework for frame semantic parsing. arXiv preprint arXiv: 10.000000. 2017.
Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. Ex- ploiting structured knowledge in text via graph-guided representation learning, arXiv preprint arXiv:2004. 14224, 2020.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. Matching the blanks: Distributional similarity for relation learning. arXiv preprint arXiv: 1906.03158, 2019.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Cohen. Open domain question answering using early fusion of knowledge bases 2018
and text. arXiv preprint
Haitian Sun, Tania Bedrax- Weiss, and William W Cohen. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. arXiv preprint arXiv:1904.0953: 2019a.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng. Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Emnie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.00 09223, 2019b.
Alon Talmor, Yanai Elazar, Yoav Goldberg. and Jonathan Berant. olmpics-on what language model pre-training captures. arXiv preprint arXiv: 13283, 2019.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition-based multi- relational graph convolutional networks. arXiv preprint arXiv: 1.03082, 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1rXiv:1710.10900 2017.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. Facts experts: Adaptable and interpretable neural memory over symbolic knowledge. arXiv preprint arXiv:2007.00844) 000449 2020.
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communi- cations of the ACM, 57(10):78 85, 2014.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al. Deep graph library: Towards efficient and scalable deep learning on graphs. arXiv preprint arXiv: arXiv: 1909.01315 2019a.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. arXiv preprint 1rXiv:I9111 001.66136, 36, 2019b.
Thomas Wolf, Lysandre Debut, Victor Sanb, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, abs//910.03771, 2019.
10

---

Under review as a conference paper at ICLR 2021
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclope- dia: Weakly supervised www.ledge-pretrainng language model. arXiv preprint arXiv:1912.0963) 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753-5763, 2019.
Yuyu Zhang, Hanjum Dai, Zornitsa Kozareva, Alexander Smola, and Le Song. Variational reason- ing for question answering with knowledge graph. arXiv preprint 1xXiv:1709.04000 2017.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv: /rXiv:9905.07/29, 2019.
A
APPENDIX
A.1 IMPLEMENTATION DETAILS
The dimension of hidden states in the knowledge module is 768, the same as RoBERTa-base and the number of attention heads is 8. During pre-training, the batch size and length of text sequences are 1024 and 512 respectively. The batch size of KG entities is 16,384. The number of training epochs JAK is optimized by Adam (Loshchilov Hutter 2017) using the following parameters: 81 0.9, 32 0.999,
1e-8, and weight decay of 0.01. The learning rate of the language module is warmed up over the first 3,000 steps to a peak value of le-5, and then linearly decayed. The learning rate of our knowledge module starts from lc-4 and then linearly decayed.
A.2 COMPUTATION ANALYSIS
The computation of the KG module is much less than the LM module. For BERT-base or RoBERTa- base, the number of inference computation flops (#flops) over each sequence (length 128) is over 22 billion [1, 2]. Here, we theoretically compute the number of flops of the KG module as follows: The sequence length N' 128, and hidden dimension is usually less than N/5. The number of sampled neighbors per entity of layers of the GNN based KG module L 2. It follows that the #flops of KG module is about N/5 x n²h x 2H x 3 billion, less than 1/7 of LM computation. If we set further reduced to about 1/30 of LM computation.
768. The number of entities in a sequence
10. And the number
the #flops can be
During pre-training, another computation overhead is entity context embedding memory update (Section 3.5): Firstly, the number of entities is about 3 million and the update step interval is 500. Thus for each step on average the model processes the description text of 3e6/500 Secondly, the length of description text is 64, much smaller than the length of input text 512, and we only use LMI (the first half of LM module) for entity context embedding generation, which saves half of the computation time compared to using the whole LM module. Thirdly, the embedding up- date only requires forward propagation, costing only half of computation compared to training pro- cess which requires both forward and backward propagation. Thus. generating context embedding of 6k entities consumes about the same number flops as training 6000 x 64/(512 2 2) z 200 input texts, much smaller than the batch size 1024. In short, the entity context embedding memory 1/5 additional computation. Note this computation overhead only exists during pre-training, since entity embedding memory is not updated when fine-tuning.
6e3 entities.
update only costs 200/1022
11


Reviews:
[{"review_id": "qcir1OVe4kw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper proposed a new language modeling pretraining method that leverages the knowledge graph information. Specifically, the paper replaces the entity embedding in one hidden layer of BERT context embedding, with the corresponding graph attention embedding that is obtained from the knowledge graph. The pretraining tasks contain not only the language related tasks (like predicting masked tokens), but also the knowledge graph tasks like entity classification or relation type prediction. Experiments on few-shot learning tasks, question answering and entity classification show better performance over other pretraining counterparts. \n\n\n# Pros\n\n- The motivation of combining knowledge graphs and unstructured text data for pretraining is valid. \n- The joint training architecture is interesting. \n- The improvement over other kg+lm or purely lm baselines is consistent. \n\n\n# Cons\n\n- Some of the designs need justification.\n- Some of the baselines/ablations might be missing. \n- The experiments have some flaws.\n\n\n# Details\n\nOverall I lean towards accepting the paper, if my concerns can be properly addressed. \n\nI like the idea of having a joint training method for both the kg embedding and text embedding. Although it encounters additional implementation/computation difficulties, it entangles the two sets of information in a coherent way. Also the pretraining has been done on a large scale, with a reasonable size of knowledge graph as the backbone. I think the paper has demonstrated a nontrivial contribution to the field. \n\nThere are several potential issues with the current paper:\n\n1. It seems the paper sacrifices too much on the efficiency, while having many heuristic approximations/designs that are un-justified. For example, a) how much would the entity embedding update affect the fine-tuning; b) how does the neighborhood sampling affect the knowledge graph embedding in the context of BERT training; \n\n2. It is not sure how each individual pretraining method affects the quality of the embedding. Also for knowledge graphs, a commonly used embedding learning method is the link prediction, where one samples positive/negative <e1, r, e2> triplets for learning. The entity category prediction or relation type prediction is less common. Would it be helpful to do the link prediction instead? \n\n3. An important baseline is missing, where one first pretrain the KG embedding and BERT embedding separately, and concat/merge them with downstream fine-tuning tasks. I’d like to see the results on the three tasks in the experiments. \n\n4. For table 2 it would be nice to include the knowledge graph based approaches that don’t leverage the pretraining, including the pullnet results, or the results that come with original metaQA paper. I don’t get the comment at the bottom of page 8 why approaches like VRN in MetaQA paper are not ‘fair’, as these models actually use less information. \n\n5. For table 1 and 2, the improvement over RoBERTa seems marginal. However for table 3 the gap is huge. Could the authors provide explanations for \n1) why use entity classification tasks? It is somewhat unfair, as the pretraining of the proposed method already involved with the entity classification tasks (although in a transductive setting); Also as I mentioned in 2, link prediction would be a more preferable task for evaluating the quality of the knowledge graph embedding; \nand 2) why is the gap so large, is it due to the issue in 1)? \n\n# Questions \n\nI’d like to hear the answers to my questions above. \n\n# Improvement\n\nI highly encourage the authors to include the necessary baselines in experiments, as suggested above. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "f15ClNZIVdq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a pretraining framework for language models and knowledge graphs. As authors mentioned in the paper, many approaches recently focused on this topic, with different variations with respect to entity embedding, initialization, over-parametrization, masked language model, fine-tuning task and etc.\n\nNovelty: The authors mentioned entity embeddings from previous works in this area mainly are computed by external resources and are injected  to the model or they are learned as parameters of the model. What about K-BERT? Doesn’t seem to have these problems. The paper doesn’t provide a comprehensive comparison with K-Bert. I think the novelty of the work may need to be further justified.\nAlso, what is the relation text or description? What kind of description do the relations have? Is it useful to use Roberta for relation embedding initialization?\n\nExperiments: \nThe paper proposes a few baselines and compares the architecture with those baselines. However, there are several other works in this area. Why not comparing with those related approaches?  As an example comparison with K-BERT?\n\nIt would be also interesting to have more baselines without language models (like the GNN in Entity Classification task)? For example, just initialize the knowledge graph by a pretrained language model and evaluate the knowledge graph on questions answering? Or even have a GNN on top of that without language model? Comparison with other knowledge graph embedding approaches on Wikipedia? Also, what if we separately pretrain knowledge graph and language model. How does affect the experiments?\n\nThere is a lack of experiments to show the Effect of number of steps in graph embedding update and also training time?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dknl2hx5_CD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a method for joint pre-training of knowledge graph and text data which embeds KG entities and relations into shared latent semantic space as entity embeddings from text. The proposed model JAKET consists of two main parts: a language module and a knowledge module. The model is pre-trained on a collection of tasks: entity category prediction, relation type prediction, masked token prediction and masked entity prediction. The proposed framework enables fine-tuning on knowledge graphs which are unseen during pre-training.\n\n\nOverall, I believe that the work on knowledge-enhanced language models to be an interesting and important area of research. However, I believe the paper is not ready for publication in its current form as (i) the demonstrated improvements obtained by pre-training with the added KG module seem minor compared to the computational overhead of having to compute entity and relation embeddings using GNNs; and (ii) experimental comparison to some relevant prior work is missing.\n\nQuestions/comments for the authors:\n\n1. One of the drawbacks of the proposed method is that it assumes entity descriptions to always be available, which might be the case for Wikidata, but it is not usually the case with e.g. standard knowledge graph completion datasets WN18 and FB15k. How would fine-tuning work on knowledge graphs that do not have entity descriptions?\n2. What does M in RoBERTa+GNN+M stand for? Is it memory?\n3. The improvements over a pure language model on few-shot relation classification and KGQA are minor, especially given the computational overhead that adding a KG module entails. The authors should include a discussion on computational overhead of having a KG module vs a pure language model.\n4. The experimental comparison to existing knowledge-enhanced language models from Section 2 is missing.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LvIG7D9u75L", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to jointly pre-train language models and representations for knowledge graphs. In particular, natural language texts (English Wikipedia) are used to train context representations, while knowledge graphs (Wikidata) train entity representations (and both depend on each other). Experiments show that the approach outperforms baseline methods on several natural language understanding tasks: few-shot relation classification, knowledge graph question answering, and entity classification. \n\nThe presented approach looks very promising, however, it also leaves several doubts. One natural ablation question is whether we could simply include the knowledge graph as plain text in training only a language model (via a similar pre-training and fine-tuning)? So, whether the additional hybrid structure involving graph convolution networks is actually necessary? (Maybe the gain in accuracy is only due to the additional information that is available in the knowledge graph?) Another natural question is why the authors have not made further experiments on other datasets and knowledge graphs? Does this approach only work for English Wikipedia and Wikidata (maybe because these two are matching extraordinarily well)? Finally, I would have expected further experimental comparisons to related approaches.      \n\nAfter rebuttal: I'm also still not convinced by the experimental evaluation. For this reason, I slightly downgraded my overall rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four knowledgeable referees reviewed this paper; one reviewer (weakly) supports accept and other three indicate reject. Even with the rebuttal, all reviewers (including positive reviewer) have concerns on unconvincing experimental results (due to missing baselines for instance). I basically agree on negative reviews that this submission fails to have enough quality considering the high standard of ICLR.
Decision: Reject