Knowledge Based Multilingual Language Model
Authors: Linlin Liu
Xin Li
Ruidan He
Lidong Bing
Shafiq Joty
Luo Si
Venue: ICLR 2022 Submitted (2022)
Keywords: Language Model
Knowledge
Multilingual

Abstract: Knowledge enriched language representation learning has shown promising performance across various knowledge-intensive NLP tasks. However, existing knowledge based language models are all trained with monolingual knowledge graph data, which limits their application to more languages. In this work, we present a novel framework to pretrain knowledge based multilingual language models (KMLMs). We first generate a large amount of code-switched synthetic sentences and reasoning-based multilingual training data using the Wikidata knowledge graphs. Then based on the intra- and inter-sentence structures of the generated data, we design pretraining tasks to facilitate knowledge learning, which allows the language models to not only memorize the factual knowledge but also learn useful logical patterns. Our pretrained KMLMs demonstrate significant performance improvements on a wide range of knowledge-intensive cross-lingual NLP tasks, including named entity recognition, factual knowledge retrieval, relation classification, and a new task designed by us, namely, logic reasoning. Our code and pretrained language models will be made publicly available.

Under review as a conference paper at ICLR 2022
KNOWLEDGE BASED MULTILINGUAL MODEL
LANGUAGE
Anonymous authors Paper under double-bling review
ABSTRACT
Knowledge enriched language representation learning has shown promising per- formance across various knowledge-intensive NLP tasks. However, existing knowledge based language models are all trained with monolingual knowledge graph data, which limits their application to more languages. In this work, we present novel framework to pretrain knowledge based multilingual language models (KMLMs). We first generate large amount of code-switched syn- thetic sentences and reasoning-based multilingual training data using the Wiki- data knowledge graphs. Then based on the intra- and inter-sentence structures the generated data, design pretraining tasks to facilitate knowledge learning, which allows the language models to not only memorize the factual knowledge but also learn useful logical patterns. Our pretrained KMLMs demonstrate significant performance improvements on a wide range of knowledge-intensive cross-lingual NLP tasks, including named entity recognition, factual knowledge retrieval, rela- tion classification, and a new task designed by us, namely, logic reasoning. Our code and pretrained language models will be made publicly available.
INTRODUCTION
Pretrained language models (PTLMs) such BERT (Devlin al. 2019) and RoBERTa (Liu al. 2019) have achieved superior performances
wide range of natural language processing (NLP) tasks. Existing PTLMs usually learn universal language representations from general-purpose large- scale corpora but lack the ability to capture world's factual knowledge. It has been shown that structural knowledge data can provide rich factual information for better language understanding. A number of studies have demonstrated the effectiveness of incorporating such factual knowledge from external knowledge bases into PTLMS (Zhang et al. 2019 Liu et al. 22020: Peterss et al. 2019 Poerner et al. 2020 Wang et al. 2021a lap. However, existing knowle mnowledee-nnnchhe LMs are all trained with monolingual knowledge graph data, which limits their applications to other languages.
In this work, attempt explore knowledge-enricher pretraining under multilingual setting, which is rarely explored in prior work. There are several motivations. First, a multilingual version of knowledge-enriched PTLM allows a larger range of population to benefit from the knowledge- enriched representations. Second, we argue that existing multilingual pretraining framework can benefit from incorporating structured multilingual knowledge. Knowledge language-agnostic, i.c. different languages share almost the same entities and relations. We can opportunisticall obtain huge amount of aligned data across different languages from existing knowledge graphs. Incor- porating this information into the multilingual PTLMs not only helps the model better capture the factual knowledge but also improves its lingull capability.
We propose KMLM, a novel framework for Knowledge- basedd Multilingual Language Model Pre- training. Unlike prior knowledge-injection approaches (Zhang al.) 2019 Liu et al. 2020 Peters et al. 2019 Wang et al. 2021a), the proposed framework requires neither structured knowledge en- coding to encode entities or relations with separate encoder, nor heterogeneous information fusion to fuse multiple types of embeddings (e.g. entities and words). The key idea of KMLM is to convert the structured knowledge from knowledge graphs to sequential data which can be directly fed as input to the language model pretraining. Specifically, we generate two types of training data the code switched knowledge data and the reasoning based data. The former is obtained by generating code switched sentences from triplets of the multilingual knowledge graph Wikidata (Vrandečić &

---

Under review as a conference paper at ICLR 2022
Krötzsch 2014). The later is constructed by converting cycles from Wikidata knowledge graph into word sequences in different languages, which contain rich logical patterns. We then propose two pretraining tasks that are operated on the code switched and reasoning based data, respectively. One designed to memorize the factual knowledge across languages, and the other is to learn the logical patterns by reasoning.
Compared to existing knowledge-enrichen pretraining methods (Zhng et al. 2019 et al. 2020 Peters al. 2019), KMLM has the following key advantages. (I) It is explicitly trained derive new knowledge through logical reasoning. Therefore, in addition to memorizing knowledge facts, also learns the logical patterns from the data. (2) KMLM does not require separate encoder for knowledge graph encoding. Furthermore, it does not rely on any entity linker to link the text to the corresponding entities, as done existing methods (Zhang al. 2019) Peters al., 2019) Poerner et al., 2020 (3) KMLM keeps the model structure of the multilingual PTLM without introducing any additional component during both training and inference stages. This makes the training much easier and the trained model is directly applicable to downstream NLP tasks.
We evaluate KMLM on a wide range of knowledge-intensive cross-lingual NLP tasks, including named entity recognition, factual knowledge retrieval, relation classification, and logic reasoning. The novel logic reasoning task is designed by us to test the reasoning capability of the models. Our KMLM achieves consistent and significant improvements on all knowledge-intensinn tasks, meanwhile it does not sacrifice the performance on general NLP tasks.
RELATED WORK
Knowledge-enhanced language modeling aims to incorporate knowledge, concepts and relations into the pretrained language models (Devlin et al. 2019 Liu et al. 2019 Brown et al. 2020). which
proved to be beneficial to language understanding (Talmor et al. 2020a).
The existing approaches can be roughly divided into two lines: implicit knowledge modeling and explicit knowledge injection. Previous attempts on implicit knowledge modeling usually consist of entity-level masked language modeling (Sun al.) 2019
[ Liu al. (2020), entity-based replacement prediction (Xiong et al. 2020). knowledge embedding loss as regularization (Wang et al. 2021b) and universal knowledge-text prediction (Sun al., 2021). In contrast implicit knowledge mod- eling, the methods of explicit knowledge injection separately maintain a group of parameters for representing structural knowledge. Such methods usually require a heterogeneous information fu- sion component to fuse multiple types of embeddings obtained from the text and knowledge graphs.
Zhang al. (219); Poerner al. (2020) employ external entity linker to discover the entities in the text and perform feature interaction between the token embeddings and entity embeddings during the encoding phase of a transformer model. [Peters al.] (2019) borrow the pre-computed knowl- edge embeddings as the supporting features of training an internal entity linker. [Wang et al. (2021a) take the advantage of adapter (Houlsby et al., 2019) and insert an adapter component in each trans- former layer to store the learned factual knowledge. Despite the effectiveness of these methods on mhowledge-nntensin
tasks, the research for jointly modeling knowledge and pretraining multilin- gual language model is still left blank. Besides, the above studies only focused memorizing the existing facts but ignored the reasoning over the unseen/implicit knowledge that is derivable from the existing facts. Such reasoning capability is regarded as crucial part of building consistent and controllable knowledge-based models (Talmor et al.) 20206}.
In this paper, with the purpose of consolidating knowledge modeling and multilingual pretraining Mulcaarr et al. 2019 Conneau et al.] 2020), and boosting the capability of implicit knowledge reasoning, explore the solutions for multilingual knowledge-enhanced pretraining.
FRAMEWORK
In this section, we describe the proposed framework for knowledge based multilingual language model (KMLM) pretraining. We first generate a large amount of knowledge-imtensive multilingual training data from the Wikidata (Vrandečié & Krötzsch, 2014) knowledge base, and then design tasks to train the language models to memorize factual knowledge and learn logical patterns from the generated data.
2

---

Under review as a conference paper at ICLR 2022
3.1 KNOWLEDGE INTENSIVE TRAINING DATA
In addition to the large-scale plain text corpus that is commonly used for language model pretraining. we also generate large amount of knowledge intensive training data from Wikidata (Vrandečid & Krötzsch 2014), publicly accessible knowledge base by collaborative editing. Wikidata is composed of a large set of knowledge graph triplets (h, r, t), where and t are the head and tail entities respectively, r is the relation type. As shown by example in Table most of the entities, as well as the relations in Wikidata, are annotated in multiple languages, and in each language, many aliases are also given, although they usually less frequently used.
Table I: An example of the Wikidata entity labels and aliases in multiple languages. This example taken from https: //t:://wwww www.. wi kidata .or.a.ogg//ooo... wwtgg/wikki /ikki///// 420 Q1420 is the unique entity ID.
ID
Language
Label
Aliases
English
motor
auto, autocar, automobile, motorcar, coche, carro, auto, automovil,
Spanish
automóvil
Q1420
Hungarian
autó
gépkocsi, automobil, személygépkoesi, személyautó...
Original (en):
(motor car, designed to carry, passenger)
Code Switched (en-fr):
motor car [mask] conçu pour transporter [mask] passenger. motor car [mask] designed to earry [mask] passager.
Code Switched (en-fr) & Alias Replaced: automobile [mask] destiné transport [mask] passenger. motor car [mask] intended to carry [mask] passager.
Figure 1: Examples of the en-fr code switched synthetic sentences. The words replaced with trans- lations or aliases are marked with bold font and underline, respectively.
Code Switched Synthetic Sentences Training language models on high quality code switched sentences is one of the most intuitive ways to learn language agnostic representations (Winata al.) 2019), where the translations words/phrases can be treated in a similar way as their aliases. Mean- while, the code mixing techniques have proved to be helpful for improving cross-lingual transfer performance in many NLP tasks (Qin et al. 20200; Santy et al. 2021). Therefore, we propose novel method to generate code switched synthetic sentences using the multilingual knowledge graph triplets. See Fig. [ for some generated examples.
For each triplet (h, r, E) in Wikidata,
hyo denote the default label of h in language I. For the entity Q1420 in Table T hen.o is "motor car" and hes. is "automóvi". h4s denotes the aliases when the integer respectively. Since English is resource-rich and often used as the source language for cross-lingual transfer, we only consider language pairs of ((en, I')} for code switching, where 1' any other none English language. With such design, English can also work as a bridge for cross-lingual transfer between a pair none English languages.
0. We define rı,s and fi,; the same way for the relation and the tail entity,
Specifically, the code-switched synthetic sentences for (h, r, t) can be generated in 4 steps: 1) Deter- mining the language pair (en, I'); 2) Finding the English default labels (hen,o: Fen,us ten,o); 3) For each item in the triplet, uniformly sample value E {true, false). if v true and the item has translation (i.e. default label) in then replace the item with the corresponding translation in l': Generate the sequence of "k [mask] [mask] 1." by inserting mask tokens. The alias replaced
3

---

Under review as a conference paper at ICLR 2022
sentences can be generated in similar way, except that we random sample aliases in the desired language to replace the default label in steps 2 and
President of the United States
Ritu Nooda
father
Raj Kapoor
sibling
official residence
position held
mother
spouse
White
residence
Barack Obama
Rajiv Kapoor
mothe
Krishna Kapoor
(a) Cycle length
(b) Cycle of length 4.
Figure 2: Examples of the cycles extracted from knowledge graph.
Reasoning Based Training Data The logical reasoning ability allows humans to solve more com- plex problems with limited information. However, this ability was not given enough attention when training the previous language models. knowledge graphs, we can use nodes to represent entities, and edges between any two nodes represent their relations. In order train the model to learn logical patterns, we generate a large amount of reasoning based training data by finding cycles from the Wikidata knowledge graph. As shown with example in Fig. 2(a) the cycles of length can be viewed as the basic components for more complex logical reasoning process. We train lan- guage models to learn the co-occurrence of the relations and entities so as infer the best candidate relations for incomplete cycles, i.e. deriving the implicit information from the given context.
Similar the structure of the code switched synthetic sentences discussed above, the cycles in Fig. [[a] is composed of 3 triplets, and hence can be converted to 3 synthetic sentences (the first example shown Fig. 4 To increase the difficulty, we also extract cycles of length 4 to generate the reasoning oriented training data. However, we find simply increasing the length of cycles will make the samples less logically coherent, so we require the cycle to have least one additional diagonal edge. Fig. [(b shows such an example. It can be converted to training sample of 5 sentences in the same way as above. For the multilingual reasoning based data, we only generate monolingual sentences, i.c. without applying code mixing.
We treat Wikidata as an undirected graph when extracting cycles. Given an entity, the length-3 cycles containing this entity be easily extracted by first finding all of the neighbour entities, and then iterating through the pairs neighbour entities to check whether they are also connected. The length-4 cycles with additional diagonal edge connecting the two neighbours can be extracted with a few extra steps. Assuming we have identified a length-3 cycle containing entity A and its two neighbour entities B and C, can iterate through the neighbours of (excluding A and C') to check whether it is also connected to C., We also need to remove duplicates when generating large amount of cycles.
3.2 PRETRAINING TASKS
Multilingual Knowledge Oriented Pretraining In the generated synthetic sentences, the "[mask]" tokens are added between entities and relations to denote the linking words. For ex- ample, the first mask token "motor car [mask] designed carry [mask) passager." may denote "is", while the second one may denote "certains" " Since the ground truth of such masked linking words are not known, we do not compute the loss of those corresponding predictions. Instead, we randomly mask the remaining tokens in the code switched synthetic sentence, and compute the cross entropy loss Los over these masked entity and relation tokens (Fig.
"certains" in French "some" "certain".
4

---

Under review as a conference paper at ICLR 2022
Cross Entropy Loss
motor
designed
Transformers N
[mask]
car
[mask]
[mask]
to
carry
[mask]
passager
Figure 3: MLM on the code switched synthetic sentence "motor car [mask] designed to carry [mask] passager.". The cross entropy loss Lcs is only computed over the randomly masked entity and relation tokens highlighted in lime green. For simplicity, the sub-word tokens are not shown in this example.
Length-3 Cycle:
Original: President of the United States [mask] official residence [mask] White House. Barack Obama [mask] residence [mask] White House. Barack Obama [mask] position held [mask] President of the United States.
Masked: President of the United States [mask] official residence [mask] White House. Barack Obama [mask] [mask]
[mask] White House. Barack Obama [mask] position held [mask] President of the United States.
Length-4 Cycle:
Original: Ritu Nanda [mask] father [mask] Raj Kapoor. Ritu Nanda [mask] mother [mask] Krishna Kapoor. Rajiv
Kapoor [mask] mother [mask] Krishna Kapoor, Rajiv Kapoor [mask] sibling [mask] Ritu Nanda. Raj Kapoor [mask]
spouse [mask] Krishna Kapoor.
Masked: Ritu Nanda [mask] father [mask) Raj Kapoor, Ritu Nanda [mask] mother [mask] [mask] [mask]. Rajiv
Kapoor [mask] mother [mask] Krishna Kapoor, Rajiv Kapoor [mask] [mask] [mask] Ritu Nanda. Raj Kapoor [mask] spouse [mask]] Krishna Kapoor.
Figure 4: Examples of the masked training samples for logic reasoning. The relations are highlighted orange. The masked entity and relation tokens are highlighted lime green.
Logical Reasoning Oriented Pretraining We design tasks to train the model to learn logical reasoning patterns from the synthetic sentences generated from the length-3 and length-4 cycles. As the examples shown in Fig. 4 we convert the relation and entity prediction tasks into the same format masked language modeling. For the length-3 cycles, each entity appears exactly twice in every training sample. If we design task to predict masked entities, can be easily solved by counting the appearance numbers of the entities. Therefore, we only mask one random relation in each sample for model training, and let the model learn to predict the masked relation tokens based on the context.
Two types of tasks are designed to train the model to learn reasoning with the length-4 cycles: 1) For 80% of the time, we train the model to predict randomly masked relation and entities. We first mask one random relation. To increase the difficulty, we also mask one or two randomly selected entities at equal chance. The lower half of Fig. 4 shows an example where one relation and one entity are masked. 2) For the remaining 20% of the time, we randomly mask a whole sentence to let the model learn to derive new knowledge from the remaining context. To provide some hints on the expected new knowledge, we keep the relation of the selected sentence unmasked, i.e., only mask its two entities. The logical reasoning task loss L, can also be computed with the cross entropy loss over the masked tokens. Note that masked entity prediction is not always non-trin non-trivial in this task. For example, when we mask exactly one entity and the entity only appears once in the masked sample, then it is easy to guess E is the masked one. In Fig. concrete example is masking the first
5

---

Under review as a conference paper at ICLR 2022
appearance of "Raj Kapoor" in the original sentence of the length-4 cycle. We do not deliberately avoid such cases, since they may help introduce more diversity to the training data.
Loss Function In addition to the pretraining tasks designed above, we also train the model on the plain text data with the original masked language modeling loss LMLM used in previous works (Devlin et al. 2019; Conneau al. 2020). Therefore, the final loss can be computed as:
== LMLLM aLLcs + fr),
where a is the hyper-parameter to adjust the weights of the original MLM and the losses for model- ing the multilingual knowledge and logical reasoning.
4 EXPERIMENTS
In this section, we first describe the pretraining details of our KMLMss Then verify its effective- ness on the wwwledge-intensive NLP tasks. Finally, we examine its performance on some general cross-lingual NLP tasks.
4.1 PRETRAINING DETAILS
Our proposed framework can be conveniently implemented on top of the existing transformer en- coder based language models like mBERT (Devlin et al. 2019) and XLM-R (Conneau et al.) 2020) without any modification of the model structure. Therefore, instead of pretraining the model from scratch, it is more efficient and cost saving to initialize the model with the checkpoints of existing pretrained models. In this experiment, we build our knowledge intensive training data in 10 lan- guages, including English (en), Vietnamese (vi), Dutch (nl), German (de), French (fr), Italian (it), Spanish (es), Japanese (ja). Korean (ko) and Chinese (zh). We only the 5 million entities and 822 relations filtered by Wang et al. (2.21b). and generate 250 million code switched synthetic sen- tences and 100 million reasoning based training samples following the steps described in $31 In addition, 260 million sentences are also sampled from the CC100 corpus B (Wenzek et al. 2020 for the 10 languages.
Our models 2 base and large checkpoints. respectively. Then we continue pretrain these models with the pro- posed tasks ($3.2). The subscript / is used to denote the models trained with the logical reasoning tasks. In addition, we also train a variant, KMLM-XLM-RRASEE on the same data but without the logical reasoning tasks, that is when training this model, the masked language modeling task is also used on the reasoning based data. Previous studies showed the original mBERT model outperforms XLM-R on the X-FACTR (Jiang et al. 2020) and RELX (Köksal & Özgäir 2020) tasks, so we also train a KMLM2-mBBB
and
are initialized with the XLM-R
model with the mBERT base checkpoint as initialization. Since the orig- inal mBERT is trained using the Wikipedia corpus, for more faithful comparison, replace the CC100 plain text with sentences sampled from Wikipedia to train MMLMR-BBERT. To speed up pretraining, we use the max sequence length of 128 to train the models first, and then finetune them on plain text with max the sequence length of 512 for another 600 steps. See Appendix $ A. for the detailed statistics of the training data and tlyper-paramedt used for pretraining.
4.2 CROSS-LINGUAL NAMED ENTITY RECOGNITION
Named entity recognition (NER) involves identifying and classifying named entities from unstruc- tured text data. Our generated training data contains a large amount of entities, and describes their relations more concisely compared with the natural sentences. This may allow the language mod- els to directly memorize these entities during pretraining and learn their similarities and differences more efficiently for better NER performance.
We conduct experiments on Con1102/03 (Tjong Kim Sang. 2002 Tjong Kim Sang & De Meulder 2003) and WikiAnn (Pan et al.) 2017) NER data to verify the effectiveness of our framework. For convenience, the WikiAnn data processed by Hu et al. (2020) is used in our experiment. We use the
Include 125 million code switched sentences and 125 million alias replaced sentences.
http: / //aata wtt.tt.tt.......................
100/

---

Under review as a conference paper at ICLR 2022
same transformer based NER model and the best models with the English train and dev splits, respectively. Finally, the NER models are evaluated on the target language test sets to check their zero-shot cross -ingual NER performance. All of the results are averaged over runs with different random seeds. The baseline model results reproduced in the same environment.
as Hu et al. (2020) finetune and select
The results on Conl102/03 data are presented in Table Compared with XLM-RBASE: our model KMLM-XLMM-RAAS improves the average Fl on all 4 languages by 1.75 points. Especially on Ger- man, our model demonstrates 4.89 points improvement. There is no obvious performance differ- ence between -EELMMXXLM--EE and KMLM MEEMMg--LM-RRA XLM-RBASE because NER may not need logical reasoning. Even when compared with XLM-RLARGE, our large model still improves the average performance by 0.92.
Table 2: Zero-shot cross-lingual NER Fl the Conl102/03 datasets.
en
de
nl
es
avg.
XLM-RBASE
91.16
68.87
79.00
76.70
78.93
KMLM XI.M-BAASE (ours)
91.38
73.76
81.55
76.03
80.68
KMLM R-LL -----XMMMRRRA- ME-RAASE (ours)
91.47
73.52
80.95
76.59
80.63
XLM-RLARGE MULLMM--XLMRRAAA (ours)
92.98
73.79
82.00
79.33
82.03
92.81
76.22
84.12
78.63
82.95
The WikiAnn dataset allows us to evaluate our models on all of the 10 languages used for pretraining. As the results shown in T our base and large models improve the average performance by 2.18 and 0.32 respectively, over the corresponding XLM-R models. We notice the results on ja and zh are much lower compared with the other languages. The result of XLM-RLARGE is even worse than XLM-RBASE. If we exclude ja and zh, the average performance improvement of our base and large models can be further increased 3.57 and 2.01, respectively.
Table 3: Zero-shot cross-lingunt NER Fl on the WikiAnn dataset.
en
V3
ml
de
fr
65
zh
avg.
Wio jaizzh
XLM-RRASE
$2.59
68.09
76.50
71.05
26.32
62.52
KMI M-XLM-RRASS (ours)
83.75
70.73
82.44
78.03
78.88
80.20
74.97
18.39
58.16
20.58
64.61
75.89
SML.M,-XLM-RR. (ours)
83,43
70.55
82.18
77.87
79.19
80.06
75.96
19.32
57.54
20.95
64.70
75.85
77.61
78.92
79.93
73.59
59.27
28.35
66.39
77.33
MMLM (ours)
85.07
77.89
84.55
81.32
83.65
82.57
78.93
14.95
69.68
19.43
66.91
79.34
4.3 MULTILINGUAL FACTUAL KNOWLEDGE RETRIEVAL
benchmark for assessing the capability of multilingual pretrained language model on capturing factual knowledge. It provides multilingual cloze-style question tem- plates and the underlying idea is query knowledge from the models for filling in the blank of these question templates. From (Jiang et al., 2020). we notice the performance of XLM-RRASE is much worse than mBERT (see Table 4). It probably because mBERT has much smaller vocabulary than XLM-R (120k vs 250k) and employs Wikipedia corpus instead of the general data crawled from the Internet. So we also pretrain <MLMg-mBEL for more comprehensive comparison. As we can see from Table 4 all of the models trained with our framework demonstrate significant im- provements on factual knowledge retrieval accuracy, which again shows that our method can help models learn more accurate factual knowledge.
X-FACTR (Jiang al. 2020)
Table 4: Factual knowledge retrieval results (acc., %) on X-FACTR.
en
es
fr
n1
ja
kn
vi
7h
avg.
MBERT
8.4
8.7
1.0
4.7
4.5
KMLM, KMLM MBBRR (ours)
13.0
10.9
8.5
11.8
2.0
3.2
10.1
10.7
8.8
XL MLM-RRASE
4.5
3.1
2.0
1.6
1.8
2.1
3.6
1.0)
2.5
RMLM-XLM-RRAAS (ours)
8.4
4.8
4.3
5.7
1.3
5.5
2.5
KMI g...M..---MM--..... MILM-R (ours)
8.6
4,8
4.2
5.6
1.6
4.2
5.8
3,0
4.7
XLM-
7.9
4.4
3.8
5.0
2.9
5.2
5.7
1.0
4.5
KMLM, XL.M-B R RLARGE (ours)
10.5
5.5
6.9
7.1
1.1
6.7
5.7
1.6
5.6
7

---

Under review as a conference paper at ICLR 2022
4.4
CROSS-LINGUAL RELATION CLASSIFICATION
RELX (Köksal & Ozgür, 2020) is developed by selecting a subset of KBP-37 (Zhang & Wang 2013). commonly-used English relation classification dataset, and by generating human transla- tions and annotations in French, German, Spanish, and Turkish. We evaluate the same set of models as 44.3 since mBERT performs better than XLM-RBASE also on this task. The evaluation script provided by Köksal & Ozgür (202)) is used to finetune the pretrained models on English training set and evaluate on the target language test sets. As the results shown Table [5] our base mod- els outperform the baselines by 0.8% to 1.6% on average. The accuracy of the large model is also increased by 0.6% on average.
Table 5: Zero-shot cross-lingual relation classification performance (acc., %) on RELX.
en
es
de
avg.
mBERT
65.8
58.9
58.5
58.2
60.3
KMLM M (ours)
64.2
59.5
59.1
61.7
61.1
XLMR
62.7
55.1
54.8
54.3
56.7
CMLM-XLM- RARA (ours)
60.0
58.0
57.5
57.8
58.3
KMLM XLM--RASS (ours)
60.3
58.7
54.4
59.4
58.2
XLMR LRRGE
62.8
62.6
60.4
59.5
61.3
KMLM XLM- XLLM-LARGE (ours)
63.5
63.7
60.1
60.4
61.9
4.5 CROSS-LINGUAL LOGICAL REASONING
Dataset We propose a cross-lingual logical reasoning (XIR) task in the form of multiple-choice questions to verify the effectiveness of our logical reasoning oriented pretraining tasks ($3.2) in an intrinsic way. An example of such reasoning question is given in Fig. [] The dataset constructed using the length-3 and length-4 cycles extracted from Wikidata, same the examples shown in Fig. 2 For each cycle, we pick a triplet to create the question and answer. The question is created by asking the relation between pair of entities in that triplet. 6 choices are provided for each question (including the correct answer), which contains all of the relations appear in the cycle and some sampled relations associated with the two entities. The remaining triplets from the eycle used as the context, which is in the form of knowledge graph. The model is required to select the most probable choice according the given context and question.
We manually annotate 1,050 samples in English to build the test set. The train and dev sets are automatically generated, and then cleaned by balancing the appearances of entities, relations and answers. After cleaning, we randomly select 3k train samples and Ik dev samples for the experiment. Then the multilingual test data in the other 9 non- non-English languages (see Sec. 4.1 are generated by selecting the entity/relation labels in the desired languages from Wikidata. The cycles used to build the test set are removed from the pretraining data, so our models have never seen them beforehand. More statistics about the dataset can be found in Appendix AA.
Context: (Poland, located in time zone, UTC+01:00) (Poland, located in time zone, Central European Time) Question: What is the relation between UTC+01:00 and Central European Time?
Answer: said be the
Choices:
said to be the same as, located in time zone, instance of, part of, has part, followed by
Figure 5: An example (English) extracted from our cross-lingual logic reasoning (XLR) dataset.
Results We modify the multiple choice evaluation script implemented by Hugging Face for this experiment. The models are finetuned on the English training set, and evaluated the test sets in different target languages. Results are presented in Ta All of our models outperform the base- line significantly. Moreover, 2MM. REASE also outperforms SMLM-XLM-PR tently though both of them are trained on the same data, which proves the usefulness of our proposed logical reasoning oriented pretraining task.
consis-
http:///.ttttca.nonnaaee//aaaa/r pptorch/multiple
8

---

Under review as a conference paper at ICLR 2022
Table 6: Zero-shot cross-lingual logic reasoning performance (acc., %6).
de
1r
k0
avg.
XLM REASE
64.38
46.38
49.36
45.46
46.38
21
32.33
54,06
39.11
27.80
42.70
KML M-XLMMRRASEE (ours)
71.52
59.60
62.73
60.03
60.12
32.
40.47
62.57
52.76
42.03
54.48
KMLM,, g. XLM REASE (ours)
71.52
63.52
64.73
66.19
63.68
37.04
43.39
63.42
53.17
45.77
$7.45
XLMM-LLARGE
79.39
68, 38
73.46
71.20
7032
56.25
47.61
70.98
66.00
55.11
65.92
KMLMM XLM Rj RLARGE (ours)
87.07
85.87
83.58
86.03
83.80
75.04
75.39
85.01
83.58
83.74
82,91
4.6 GENERAL CROSS-LINGUAL TASKS
Our models are directly trained on the structured knowledge graph data. Though we attempt to min- imize its difference from the natural sentences when designing the pretraining tasks, it unknown how the difference affects cross-lingual transfer performance on the general NLP tasks. Therefore, also evaluate our models the part-of-speech (POS) tagging, question answering and classifica- tion tasks prepared by XTREME (Hu et al.) 2020). Experimental results are shown in Table 7 Note that many of the languages covered by these tasks are not in our pretraining data, but we include all their results when computing the average performance. Overall, the performance of our models is comparable with the baselines on all of the tasks, except POS. because the POS task more sensitive to the change of the training sentence structures, while the impact the other tasks not obvious. From these results we can say that our KMLMs achieve consistent improvements on the knowledge-intensnn tasks, as shown by the experimental results in the previous subsections, without sacrificing the performance on the general NLP tasks.
Table 7: Zero-shot lross-lingual POS, QA and classification results. Note that the performance of the languages not appearing prepared pretraining data also counted.
POS
XQuAD FI/EM
MLQA F1/EM
TyDi QA FI/EM
XNLI
PAWS-X
Metrics
Acc.
Acc.
XLM-RBASE
73.71
69.6/53.9
64.9/47.2
45.2/
73.66
84.80
KMLM-XLM KLM-REASE (ours)
72.69
69.5/534
65.4/47.3
49.4/30.9
73.39
84.47
KMLM XLM--RASE (ours)
72.76
63.2/53.3
64.7/47.0
48.7/29.2
73.56
85.07
XLMM-RARGE
75.25
76.8 / 60.9
52.5/54.
66.6/46.6
78.95
87.81
KMI M g- XLLM-M LARGE (ours)
73.52
76.5 / 60.6
72.07 53.7
66.4 47.9
78.55
87.66
CONCLUSIONS
In this paper, we have presented a novel framework for knowledge-basee multilingual language modeling. Our approach firstly creates synthetic multilingual corpus from the existing knowl- edge graph and then tailor-makes two pretraining tasks, namely, multilingual knowledge oriented pretraining and logical reasoning oriented pretraining, to consolidate the knowledge modeling and multilingual pretraining and boost the capability of implicit knowledge modeling. We evaluate the proposed framework on a series of tnowledge-intensie cross-lingual benchmarks and the compar- ison results consistently demonstrate its effectiveness.
REPRODUCIBILITY STATEMENT
We propose framework to pretrain a knowledge- based multilingual language model. All of the data used in our framework, including the Wikipedia/CC100 plain text and the Wikidata knowledge base, can be conveniently accessed from the Internet. Detailed hyper-parameter statistics are presented in the Appendix. Except the self-constructed cross-lingual logical reasoning dataset, all of the downstream task datasets and evaluation scripts used in our experiments can be easily obtained by either following the provided URL checking the cited papers. Our code used for model pretraining can be found in the supplementary material. We will also make the code, the pretrained language models and all data prepared by us publicly available.
and training data

---

Under review as a conference paper at ICLR 2022
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pp. 1877-1901, 2020. URL 970
ngs neur ciuurrpss.. .ts.c//
Alexis Conneau, Kartikay Khandelwal. Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451 2020. https: main. 747
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, MAMACI-HLI..p.4177 4186, 2019. (tRLhttps:/// /atp:::///aaattt/ hoggy..ooooo.oooooooooooo.oooo.. . /rg/N19-g-- --119119232
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer leam- ing for NLP. In Proceedings of ICML, volume 97, pp. 2790-2799, 2019. URL https:
//proceedings... mll press / /////ouull. aby1 9a html
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In Proceedings of ICML. pp. 441 11 1421, 2020. URL http:://proced ngs mmlr.
http:hh22bbpppp. pdf
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. X- FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Pro- ceedings of EMNLP, pp. 5943-5959, 2020. URL https::/ httppss//acaaathhlogg org./020.
emr.p-main. 479
Abdullatif Köksal and Arzucan Özgür. The RELX dataset and matching the multilingual blanks for cross-lingual relation classification. In Findings of EMNLP, pp. 340-350, 2020. URL https:
wttp://oooooy..///220...... //aclanthology. owq///20.0oo./oooooo ioteetoffiiiiinnn findings-enni emmlp.mnnpp...mm
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-BERT: Enabling language representation with knowledge graph. In Proceedings of AAAI, 2020. URL
https://oje aaai. www.oorg/neeeeee http:///s..aaa..o/iii...e..... http:///////aaaaarra////////////
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.116990 2019. URL httltt///////...cco wwtpa/wwwww..r.///////aa///////. 11692
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. Polyglot contextual representations improve crosslingual transfer. In Proceedings of NAACL-NLT, pp. 3912-3918, 2019. URL https:
7/ 7/www. www.waclweb.co org/ anthology/N11 1392
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1946-1958, 2017. URL https::/acc wwweelannnhooooooo logy. wwtppoggg/P1111111
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. Proceedings of EMNLP, pp. 43-54, 2019. URL / org/p1rrg111 1005
10

---

Under review as a conference paper at ICLR 2022
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. E-BERT: Efficient- entity embeddings for BERT. In Findings of EMNLP, pp. 803-818, 2020. URL https://
aclanthology org/2020. wttp:/222...inniii.. www.ings-emnn.........
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP. In Proceedings of the Twenty-Ninie Inter- national Joint Conference Artificial Intelligence. IJCAI 2020, pp. 3853-3860, 2020. URL
/ttp:////aaa..2//33333333 2020/533
Sebastin Santy, Anirudh Srinivasan, and Monojit Choudhury. Bertologicomix: How does code- mixing interact with multilingual bert? In Proceedings of the Second Workshop on Domain Adaptation for NLP, pp. 111-121, 2021. URL
org/2021,
11tttnnpp-111..101 pd
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhangg Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Emiee: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.090 09223,, 2019. URL httthttps:/aaat
org/ httg//ab///e 1904 09223
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang. Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang. Peng Sun, Wei Liu, Xuan Ouyang. Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. 2021. URL htttttspp//axix> org. abs/ / 211. 02137
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. oLMpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758. --8, 2020a. UBL/:ttteee//a...attooooooooooot
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap-of-thought Teaching pre-trained models to systematically reason over implicit knowledge. In Proceedings of NeurlPS, pp. 20227-20237, 2020b. URL http://///Ruiipppp///////e//e/aa
wwwpeediiinnnnunninnn.. neur iss. cc cc/paper/ pddf
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-indeeeendet named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-20022, 2002.
htttps::/accaathhlloge..
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACI. 2003, pp. 142-147, 2003. URL. https:
7/ wwwptthhoooooy..g/oo11-30-
Denny Vrandečić and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. Commun.
ACM, 57(10):78-88 85, September 2014. URL httlhtttt///oool www.:/i..rrg/1..1 11..g/g...11.... /111//22//293333
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of ACL, pp. 1405-1418, 2021a. URL https:: seteps://cclanthoogg
org/2021. findings 121
Xiaozhi Wang, Tianyu Gao. Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Rep- resentation. URL Transactions of the Association for Computational Linguistics, 9:176-194, 2021b.
00360
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave CCNet: Extracting high quality monolingual datasets from web erawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pp- 4003-4012, 2020. URL
1re.-1....... 1.494
Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Code-switched language models using neural based synthetic data from parallel sentences. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 271 271-280. Association for Computational Linguistics, 2019. ht..ttttt:/// //ac:::://ccccco antho www.looyyyyy....o... org/ -
11

---

Under review as a conference paper at ICLR 2022
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: Weakly supervised knowledge-pretrainn language model. In Proceedings of ICLR, 2020. URL
lttpp::/cpenreview net
Dongxu Zhang and Dong Wang. Relation classificatton via recurrent neural network. arXiv preprint abs 1508. 01000
arXiv:1508.01006, 2015.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. In Proceedings of ACL, pp. 1441-1451, 2019.
URL https://aclanthor ogt.oo/oo//o1111/.3033
A APPENDIX
A.1
LANGUAGE MODEL PRETRAINING DETAILS
Training Data
The statistics of the data used for pretraining shown in Table 8
Table 8: Statistics of the data used for pretraining.
Description
Value
number languages
code switched synthetic sentences
246, 146,783,693
number of unique relation combinations in length-3 cycles number of unique relation combinations in length-4 cycles number of reasoning based training samples from length-3 cycles
29,819
239,966
24,142,272
number of reasoning based training samples from length-4 cycles
73,881,425
number of sampled CC100 sentences (KMLM-XLMM) only) number of sampled Wikipedia sentences (KMLM- -BERT only)
260,000,000
153,011,930
Hyper-Paranee Papameaarssstrss The hyper-panee paremeerreeterss used for language model pretraining presented in Table 9 After pretraining, we finetune the models on the plain text data with max sequence length of 512 for another 600 steps.
Table 9: Hyper-parameters used for language model pretraining.
Hyper-parameter
Value
learning rate
5e-5
weight decay
0
optimizer
Adam W
number of train epochs
batch size for the natural sentences batch size for code switched knowledge data
9,600
9,600
batch size for reasoning data
9,600
mim probability
0.15
max sequence length
128
number of warmup steps knowledge task loss weight (a in the loss function)
100
0.3
12

---

Under review as a conference paper at ICLR 2022
A.2 CROSS-LINGUAL LOGIC REASONING DATA
Statistics of the self-constructed cross-lingual logic reasoning (XLR) dataset are presented in Ta- ble T0 The multilingual test data in the 9 non-English languages are generated by selecting the entity/relation labels in the desired languages from Wikidata. So the statistics for their test sets are the same as English.
Table 10: Statistics of the self-constructed cross-lingual logic reasoning data (English).
Description
Value
number of samples in the train set
3,000
number of samples in the dev set 1,000
number of samples in the test set train set unique relation combinations
1,050
1,419
dev set unique relation combinations
746
test unique relation combinations
444
13


Reviews:
[{"review_id": "uES1rfn2y0q", "reviewer": "Reviewer_cZgu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths of the paper:\n\n- This addresses an important area in large language models, of better encoding knowledge in multilingual models. I believe this is a useful direction and will be of interest to the NLP community at ICLR.\n- I think the idea of generating code-switched synthetic sentences from KG triples to be really nice. If it could be shown that this technique can be used to boost performance of low-resource languages by leveraging KG resources from English, I believe this technique could be of practical use. It may also motivate us to contribute aliases to Wikidata for low resource languages.\n- The idea of the edge/relation completion task is also quite interesting. This task seems related to a few-shot/multiple choice relation classification task  but where the task presents 2 related pieces of information (KG triples), instead of a single (natural language) triple.\n \n### Weaknesses of the paper:\n\n- My main criticisms of this work stems from a single point, which is, after reading the paper, it was not exactly clear what the main problem being addressed. Although the techniques and experiments presented were thematically related, in that they used synthetic sentences derived from WikiData, it was not quite clear that they were addressing a specific problem. For example: the code-switched data augmentation idea seems to be related to the question “can we better perform knowledge transfer between languages using multilingual KG” or perhaps “using multilingual KG can augment knowledge in LMs since certain knowledge is not covered in Wikipedia/Web documents.”, or some other specific hypothesis?  Given a better defined hypothesis, it may be easier to evaluate how much of the problem is solved by the techniques in this paper.\n\n  The “logical reasoning” task, OTOH, seems to be somewhat independent of cross- or multi- linguality? Is the hypothesis that the logic reasoning task allows LMs to better perform edge/relation prediction? Is there anything specifically cross-lingual about learning this task, or would one be able to learn/evaluate its effectiveness in a mono-lingual setting?\n\n\n- Related to the above: the cross-lingual experiments show relatively modest performance wins across the different tasks evaluated. It raises the question of whether there is something specific about code-switching the triples that are helpful or whether the extra KG data, even in mono-lingual format, would have been sufficient to improve performance. Given that previous works have shown that multilingual LMs are able to transfer tasks across languages,  I think it would have been very useful to train a baseline that contains all the KG triples used in the KLML-variants, but simply with monolingual versions of the triples.\n\n  As an example of this, see the results from X-FACTR for the English language which saw the largest improvement in performance. It seems that the improvement may have come purely from allowing the model to train on KG triples, and is potentially unrelated any cross-lingual transfer.\n\n### Other comments:\n\n- While I thought the “cross-lingual logical reasoning” pretraining task to be an interesting one, I do believe the evaluation needs a bit more work. The current evaluation, as presented in Section 4.5, is quite close to the pretraining task in both content and format. As such, it feels as this result is not as “zero-shot” and it's a bit hard to conclude whether gains are obtained from some cross-lingual transfer or whether its from a large overlap in pretraining and evaluation tasks.\n\n- Also related to the logical reasoning task: besides the intrinsic evaluation in Section 4.5, none of the other evaluated tasks saw much improvements. Is there an expectation that this pretraining objective could be really helpful on some other task? Which ones?\nIn a few of the results tables, particularly ones marked as “zero-shot” it is unclear what exactly the training setup is. For example Tables 2 and 3 claim to be zero-shot NER, however the text describes supervised training with Conll 02/03 in English. Is English in Table 2 still zero-shot or supervised? What about Table 3 (WikiAnn)? Is the model trained on CoNLL and evaluated on WikiAnn? Or is any WikiAnn training data used?\n\n- For X-FACTR it's not clear what is the training/fine-tuning setup. It doesn’t say which languages are used for training.\n\n- For RELX, we see the same problem as NER: it seems that the model is trained on English and evaluated on other languages? Please make this clear.\n\n- For XLR, please also make a note on the table that English is *not* zero-shot, but fully supervised.\n\n- Also for the XLR it was unclear how exactly the multiple choice task was formulated. While there is a reference to a HuggingFace script, it is a bit obscure (and the script can be removed at any time). It would be better to simply describe how the multiple choice task is modeled here.\n\n- Perhaps a more direct evaluation of improved knowledge representation in the KLML model is the “Multilingual LAMA” task from :\n    1.  “Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models” https://aclanthology.org/2021.eacl-main.284.pdf , https://github.com/norakassner/mlama \n\n- Also, related to training on KG triples, I think it would have been useful to compare with the work of verbalizing KG triples as in:\n  1. “The WebNLG Challenge: Generating Text from RDF Data” https://aclanthology.org/W17-3518/  , https://webnlg-challenge.loria.fr/challenge_2020/ \n  2. “Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training”, https://aclanthology.org/2021.naacl-main.278.pdf , https://github.com/google-research-datasets/KELM-corpus \n", "summary_of_the_paper": "This paper introduces two new techniques for pretraining large multilingual language models focusing on knowledge intensive tasks. It first proposes generating code-switched synthetic sentences derived from triples from a knowledge graph (WikiData) and subsequently using these synthetic sentences as input to a multilingual masked language model. In addition, the work proposes using cycles in knowledge graph nodes, it proposes a new task that uses the derived synthetic sentences and asks the model to predict (and learn from) a masked-out relation (edge). The paper demonstrates the efficacy of the new pretraining methods using a new cross- and mult- lingual tasks, including: NER (Conll 02/03), fact retrieval (X-FACTR), relation extraction (RELX), and some XTREME tasks. In addition, they describe a new cross-lingual logical reasoning task (XLR) to verify the effectiveness of the “logical reasoning” pretraining task.\n", "summary_of_the_review": "This addresses an important area in large language models, of better encoding knowledge in multilingual models. I believe this is a useful direction and will be of interest to the NLP community at ICLR. I think the two core techniques proposed here (code-switching using KG triples and using KG cycles to teach models about transitive relations) are new and interesting. The main drawback of this work is that the improvements on the tasks chosen are quite modest. It is also unclear whether a simpler version of the proposed technique (discussed above, of training using monolingual triples) would have been sufficient for these modest gains. I think the work would benefit from more clearly defining the main hypotheses in the study and more carefully evaluating (with also qualitative analysis) the benefits of the proposed techniques.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "o7nbmFPxQuD", "reviewer": "Reviewer_pMnq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-written and well-presented.\n2. The experiments are extensive and convincing.\n3. The model shows strong performance on a wide variety of tasks that is knowledge-intensive.\n\nWeakness:\n1. The experiments done are mainly in a zero-shot setting, while impressive, there might be concerns regarding finetuning performance compared with existing baselines.\n2. The idea of code-switching in multilingual knowledge is not completely new, for example, X-Factr is also doing code-switching in the prompts to better retrieve factual knowledge. This should be further clarified in the paper contribution.\n3. I think the sampling of synthetic data should be discussed more, since this is the focus of the paper pretraining method. More ablation on how to sample and handle the problems of distribution mismatch between true texts vs. template texts could shed light on what setting such pretraining would be most useful.\n4. Some focus should be given to more diverse sets of languages, esp. low-resource languages. Since it's a multi-lingual model and utilizes code-switching, I wonder if it can increase performance for some low-resource language tasks by leveraging connections to popular languages.", "summary_of_the_paper": "The paper presents a new pre-trained language model that works on multiple languages. The language model pretraining has a special focus on incorporating knowledge, by generating a large amount of code-switched synthetic sentences and reasoning-based multilingual training data using the Wikidata knowledge graphs. The language model not only memorizes the factual knowledge but also learns useful logical patterns. Experiments on a wide range of knowledge-intensive cross-lingual tasks show performance improvemnents. The author also proposes a new task to test the model's ability of logical reasoning.", "summary_of_the_review": "The paper is having really good performance across multiple tasks and is simple to implement and follow. The multilingual is a nice addition to existing knowledge-based LMs, but there are some concerns regarding the pretraining data generation and low-resource setting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dKILBtBtn87", "reviewer": "Reviewer_Rt8N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The major contribution of this work is the way of constructing synthetic sentences to encode multilingual knowledge. The code-switching is straight-forward and common in the cross-lingual and multilingual setting. And encoding logical patterns into LMs is an interesting problem and this work sampled simple “circles” from the KGs. Although sampling subgraphs is a good idea, this work’s method is still simple and does not give a clear definition of “circle”. Why are two kinds of “circles” important for logical reasoning? Could it extend to more patterns? So far, this work did not give a paradigm of formulating logical patterns from the KGs (if it is the major strength), which a bit hurts the novelty. Another weakness is that the writing is not clear. For the pre-training tasks,  what is the cross entropy loss of “L_{CS}” over the masked entity and relation tokens? And the same for the loss of “L_{L}”\n\nSome clarification questions:\n1). In all the result tables, what are the results of KMLM-XLM-R_{large}(ours)? Seems the authors did not report all the performance of KMLM-large without logical reasoning. \n\n", "summary_of_the_paper": "     This paper proposed a new pretraining framework of incorporating multilingual knowledge into language models. The LMs are pre-trained over a large amount of synthetic code-switched sentences sampled from the multilingual KG, wikidata. Besides factual knowledge, the methods also sampled subgraphs from KG and learned logical patterns. The experiments show that the framework improved many knowledge-intensive tasks.\n", "summary_of_the_review": "This work proposed to solve interesting problems of multilingual knowledge pretraining. The method is straight-forward and the pre-training task or logical patterns are not clearly written. Some ablation studies should be further investigated such as with/without normal sentences from CC100 corpus. More importantly, the author should think more about logical reasoning.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OQv6KFJgRBl", "reviewer": "Reviewer_RyT7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength of the paper:\n\nThe paper presents a framework to pre-train Knowledge-based Multilingual Language Models, the proposed model is evaluated on a wide range of knowledge-intensive tasks,  the knowledge-intensive training data is built on 10 languages.\n\nThe proposed model doesn’t require training from scratch, it can be initialized with checkpoints of existing pre-trained models.\n\nExtensive experiments are performed to justify the advantage of the proposed approach;\n- Cross-lingual NER: The proposed model improves F1 performance by an average of 1.75 points compared to XLM-RBASE which is an existing SOT (Table 2)\n- Multilingual Factual Knowledge Retrieval: Results in Table 4 shows that the proposed method significantly improves the performance compared to other methods.\n- Cross-lingual Logical Reasoning: Results in Table 6 show that the proposed model significantly outperforms the baselines. \n\nWeakness of the paper:\n\nThe technical contributions are not explicitly highlighted, the description of the proposed approach could be made more sound, and or easy to follow with either algorithm or figure/workflow.\n\nGeneral Cross-lingual Tasks: The results in table 7 demonstrate that the performance of the proposed method is less compared to existing methods.\n", "summary_of_the_paper": "The paper argues that existing knowledge-based language models are all trained on monolingual knowledge graph data, hence limiting their application to more languages.\n\nTo overcome this, the paper presents a framework to pre-train Knowledge-based Multilingual Language Models  (KMLMs).\n\nThe proposed framework (KMLMs);\nFirst generates synthetic sentences (code-switched) and reasoning-based multilingual training data using Wikidata knowledge graph.\nThen design pretraining tasks to facilitate knowledge learning, which allows the language models to not only memorize the factual knowledge but also learn useful logical patterns based on the intra- and inter-sentence structures of the generated data. \n\n KMLM is evaluated on a wide range of knowledge-intensive cross-lingual NLP tasks, including named entity recognition, factual knowledge retrieval, relation classification, and logic reasoning.\n\n It achieves consistent and significant improvements on all knowledge-intensive tasks, and it does not sacrifice the performance on general NLP tasks.\n", "summary_of_the_review": "The paper deals with Knowledge-based Multilingual representation which is relevant and a valuable contribution to the field of language understanding.\n\nThe authors provide good and clear motivation and introduction to their research work. In general, the paper is well structured.\n \nThe discussion of related work is adequate with a number of recent citations. It could be extended in the final version, especially regarding the future directions.\n\nThe evaluation part contains interesting implementation details, used technology is state-of-the-art. The evaluation is convincing and nicely shows the merit of the approach.\n\nAlthough the proposed method looks incremental, it has the following advantages:\n\n1. The proposed model is explicitly trained to derive new knowledge through logical reasoning in addition to memorizing knowledge facts.\n\n2. It does not require a separate encoder for knowledge graph encoding. And also doesn't rely on any entity linker to link the text to the corresponding entities, as done in existing methods.\n\n3. It also keeps the model structure of the multilingual pre-trained language model without introducing any additional component during both training and inference stages.\n\nRecommendation \n\nThe technical contributions could be explicitly highlighted, the description made more sound, and or easy to follow with either algorithm or figure/workflow.\n\nSection 3.1 Reasoning Based Training Data: The authors mention that “Fig. 2 (a) the cycles of length 3 can be viewed as the basic components of complex logical reasoning process”. This is however not clear, a more clear explanation would be helpful. \n\nEnglish needs some proofreading.\n\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Iz0dqH9a7f1", "reviewer": "Reviewer_C6ht", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper presents a novel way of implicitly adding knowledge base information at pre-training time that may improve performance on knowledge related downstream tasks, it is hard to assess the impact of the approach, due to a shallow evaluation on the experiments. \n\nEach of the experiments follows the same structure: the model is fine-tuned on English for the downstream task and zero-shot on the other languages. Therefore, the reported results for English are not zero-shot, and should be compared to the previous work that reported on the same benchmarks for supervised English. These are missing in all experiments. Moreover, there are other state-of-the-art models and approaches that could be evaluated on the same premise, by training on English and zero-shot evaluating on the other languages, including knowledge-enhanced ones cited in the paper, which would enable a fair comparison of the proposed approach with previous work. \n\nThe evaluation presented seems to point that including extra knowledge-based sentences to the pre-training phase helps in knowledge intensive downstream tasks. This is expected and the paper shows it when compared to vanilla pre-trained Transformer models. It is just hard to assess how impactful this addition is when compared to previous work that also, either implicitly or explicitly, incorporates knowledge from knowledge bases, as those described in the related work. While it is true that most of them were monolingual, they could be compared to the supervised English experiments in the paper, or evaluated using the same zero-shot setup.\n\n", "summary_of_the_paper": "This paper presents a new language model that is enhanced with knowledge from Wikidata. The authors use two masked language modeling strategies by training with language-switched relation triples and Wikidata graph cycles (unrolled as sequences of triples). Experiments are performed in several tasks: cross-lingual NER, factual knowledge retrieval, relation classification, logical reasoning and other tasks. ", "summary_of_the_review": "An interesting idea, but with an evaluation that does not demonstrate the true impact of the proposed knowledge-enhanced model.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the important problem of adding structured knowledge (in this case from Wikidata) to pretrained language models. The reviewers do not see this paper as ready for ICLR and recommend a number of revisions. Unfortunately the authors did not respond during the author response period. The area chair hence agrees with the reviewers.
Decision: Reject