Multilingual Pre-training with Universal Dependency Learning
Authors: Kailai Sun
Zuchao Li
hai zhao
Venue: NeurIPS 2021 Poster (2021)
Keywords: natural language processing
multilingual pre-trained language model
universal syntactic dependency parsing

Abstract: The pre-trained language model (PrLM) demonstrates domination in downstream natural language processing tasks, in which multilingual PrLM takes advantage of language universality to alleviate the issue of limited resources for low-resource languages. Despite its successes, the performance of multilingual PrLM is still unsatisfactory, when multilingual PrLMs only focus on plain text and ignore obvious universal linguistic structure clues. Existing PrLMs have shown that monolingual linguistic structure knowledge may bring about better performance. Thus we propose a novel multilingual PrLM that supports both explicit universal dependency parsing and implicit language modeling. Syntax in terms of universal dependency parse serves as not only pre-training objective but also learned representation in our model, which brings unprecedented PrLM interpretability and convenience in downstream task use. Our model outperforms two popular multilingual PrLM, multilingual-BERT and XLM-R, on cross-lingual natural language understanding (NLU) benchmarks and linguistic structure parsing datasets, demonstrating the effectiveness and stronger cross-lingual modeling capabilities of our approach.

Multilingual Pre-training with Universal Dependency Learning
Kailai Sun", Zuchao Li", Hai Zhao"
DDpaatteen of Computer Science and Engineering, Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University (kaishu2. O, charlee) @u. edu. en, zhaohai@cs sj tu. edu. cn
Abstract
The pre-trained language model (PrLM) demonstrates domination in downstream natural language processing tasks, in which multilingual PrLM takes advantage language universality to alleviate the issue of limited resources for low-resource languages. Despite its successes, the performance of multilingual PrLM is still un- satisfactory, when multilingual PrLMs only focus plain text and ignore obvious universal linguistic structure clues. Existing PrL.Ms have shown that monolingual linguistic structure knowledge may bring about better performance. Thus pro- pose novel multilingual PrLM that supports both explicit universal dependency parsing and implicit language modeling. Syntax in terms of universal dependency parse serves as not only pre-training objective but also learned representation in our model, which brings unprecedented PrLM interpretability and convenience in downstream task use. Our model outperforms two popular multilingual PrLM, multilingual-BERR and XLM-R, on cross-lingual natural language understanding (NLU) benchmarks and linguistic structure parsing datasets, demonstrating the effectiveness and stronger cross-lingual modeling capabilities our approach.
Introduction
The pre- pre-trained language model (PrLM) such as BERT [0]] and many kinds of its variants [2] B. [] have proved their effectiveness in many downstream natural language processing (NLP) tasks in-
cluding semantic textual similarity [B]], question answering [[6], sentiment classification [[Z], linguis- tie structure [4, 8] and so on. Most of these PrLM are aimed at languages that with a large amount of available linguistic resources and widely used, such English. However, it is not realistic to train an individual PrLM for all languages, especially for those low-resource languages. As a result, several multilingual PrLMs which take advantage of language universality have been published and shown good cross-lingual performance on several NLP tasks.
Despite its successes, the unsupervised method typically used by multilingual PrLMs makes cross- lingual transfer inefficiency and keeps the learning still challenging. Improvement can be made by adding explicit cross-lingual signals including bitext (XLM) [X] and word translation pairs from dictionary [IU]. This suggests that the effectiveness of multilingual PrLM can be further improved by integrating explicit universal linguistic characteristics. Existing PrLMS (III [2] have tried to incorporate monolingual linguistic structure knowledge to improve the performance across multi-
ple linguistics tasks by Multi-Task Learning (MTL) [K]]] However, the combination of universal
"oorrespnddigg author: These authors made equal contribution. This work supported by Key Projects National Natural Science Foundation China (U1836222 and 6173301 Project from Chinese National Key Laboratory of Science and Technology on Information System Security.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

linguistic structure knowledge has not been explored in the multilingual area. Learning universal knowledge across languages is more complex than learning monolingual knowledge, so a better integrating method than MTL needs to be explored.
Syntactic dependency parsing disclosing syntactic relations between words in sentence, has been found to be extremely useful for many NLP tasks [IEL, 16, [[]]. The syntactic dependency parsing also limited by low-resource languages. To meet the huge demand for training syntactic parser among various languages, the project of universal dependencies (UD) Treebanks was launched [[Z]] which provides a uniform syntactic parsing structure for different languages. Therefore, UD offers an excellent universal structure characteristic, which is worth exploiting for the multilingual PrLM.
In this paper, propose multilingual PrLM that supports both explicit universal dependency parsing and implicit language modeling. Unlike using MTI. in monolingual works, we embed a parsing scorer in our PrI.M, and directly optimizes this scorer and the encoders below it with UD) pre-training: meantime, we propose a structural encoder to encode the predicted structure given by the parsing scorer and integrated it into the final representation for other pre-training or downstream training process. Our approach can be smoothly applied to a variety of multilingual PrLM such as Multilingual-BER (m-BERT) [M]] and XLM-R [[S].
To verify the cross- pooss-lingual modeling capabilities of our model, we carry on the experiments on both cross-lingual NLU benchmarks: XNLI and XQuAD, and linguistic structure parsing datasets: UD3 v2.7, SPMRL' 14 [19), English Penn Treebank (PTB) 3.0 [20] and the Chinese Penn Treebank (CTB) 5.1 [2I]]. Our empirical results show that universal structure knowledge learnt and integrated can indeed help the multilingual PrLM obtain better universal linguistic word representations and outperform m-BERT and XLM-R baselines in all the above tasks.
2 Related Work
Monolingual PrLM with structure learning Previous works have tried to improve monolingual
[5] Some of them use plain text to implicitly learn structural knowledge, such as StrucBERT [2). which lets the PrLM encode the dependency between sentences and words by adding the word-level ordering and sentence-level ordering objec- tives during pre-training. However, this may not be suitable for multilingual setting. Since there are obvious differences in the word order of different languages, putting disordered plain text of multi- ple languages together for common training will lead to confusion in implicitly learning structural information. Contrastively, we use annotated syntactic structures knowledge UD that provide clear guidance for improving cross-lingual representation in multilingual PrI.M.
PrLM by learning linguistic structure [4, IZ 22
25
24
Other works use annotated syntactic knowledge as the structural knowledge such LIMIT-BERT [122] and LISA [24]] LIMIT-BERT is a monolingual PrL.M that achieves advancement in several parsing datasets and NLU tasks by performing MTL on multiple linguistic tasks including syntactic parsing. The main difference between LIMIT-BERT and our method is that we do not regard parsing as a pre-training objective, but use the parsing component as an intermediate structure to explicitly learn and extract of the syntactic structure, thereby reducing the black box characteristics of the model. LISA Transformer model that uses syntax information enhance SRL. It incorporates syntactic information by training one attention head predicting syntactic dependency arc. First of all, LISA only aims one specific task, so doubtful whether its method can be applied to the PrL. and improve the performance of multiple tasks. Secondly, LISA can only use the are feature of syntax but not the relations, while our approach of syntactic structural integration can integrate both of the are and relation features learned by our model into the final universal representation. By the way, our PrLM can be used directly as a well-behaved parser, but LISA clearly does not.
Multilingual PrLM with explicit cross-lingual signals Cross-lingual pre-training tasks includ- ing cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model can take advantage of bitext to learn the mappings among different languages from more perspectives [26]]. Recently, Ahmad et al. [27] proposes a multilingual PrLM training mBERT using an auxiliary objective to encode the universal dependency tree structure that helps cross-lingual transfer. However, our work significantly differs from theirs. We regard parsing not only as pre-training objective, but also an active part of the overall PrLM structure, so this part

---

US Learning Layer
L
Multi-layer Transformers
S
US Integration Layer
Token Embedding
Position Embedding
[CLS]
[SEP]
A
[CLS] X [MASK] X
[SEP]
Masked LM
M.M
Figure 1: The model architecture for UD-PrLM.
of the structure will further exert its effectiveness in downstream tasks rather than discarding it in the fine finettuning stage, which increases the adaptability from pre-training fine-tuning phase.
Universal Dependency as Language Modeling
In this work, we chose UD parse as our universal structure knowledge. Our model includes five modules: token representation, multi-layer Transformers, universal structure learning (USL) layer, universal structure integration (USI) layer and pre-training objectives. Figure shows the full model architecture of our method.
3.1 Token Representation
Take BERT [I]] as an example, in the token representation layer, given input sentence X, the sentence concatenated with two special tokens "[CLS]" and "[SEP]": [CLS], 22 .., TN, [SEP], and [CLS] is also used as the dummy ROOT node in UD training process. The input X is mapped into sequence of input embedding vectors [e1, e2.... Exxil, one for each token, which is sum of the corresponding word and positional embeddings.
Since the UD parse tree is annotated at word-level and the input sequence X in PrLM is based on subword tokenization, in order to sufficiently improve the representation of tokens and adapt to the subword-level tasks, we propose conversion strategy that extends the parsing tree from the word level: Y to the subword level: Y. and use the subword-level parsing tree as the training objective for USL. Detailed strategy description is shown in Appendix [A.I.
3.2 Multi-layer Transformers
The multi-layer Transformers architecture in model is adapted from Vaswani et al. a2.] which transforms the input embedding vectors into a sequence contextualized representation vectors H [h,, .h,,... hxi]] shared across different tasks. We Transformer architecture with L layers, A self-attention heads for each block and hidden size H:
H L) = Transformers(Embl (mb + PosEnccdingg(X)

---

Algorithm 1: Training Process
Input: MLM training data XMLM, UD Treebanks (XuD, YUD), Parameters: (p, 7, w, 0), Probability of training USL: p.
D XUD. YUD <- Token ToMMn U XUD), Token(XUD), Strategy Token)
2 Initialize 8g randomly;
for t <- to do
L 0+t1 < C + nr XtD; YUD) + LMLM (pr di, DM
5 for t < m to m do
if rrrandom.nifomm
then
O++1 < Opt(O,, CUSL(pp + nr XiD YUD) + CULM + + + de, D DMLM))
else
B.t1 t Opt
M.M (po + 7 + we + Φ1. D DMaM));;
Output: On.
3.3 Universal Structure Learning
Our USL layer follows the state-of-the-arr graph-based deep biaffine dependency parser [22]. We replace the BiLSTM encoder with the multi-layer Transformers architecture and use the hidden state of its last layer as the output of encoder H (.) = [11"), (L) _n ,..,,,[[]]]
(L))
For both are and label predictions, two separate MLPs are used to distinguish two kinds of low- dimensional vectors as head and dependent representations respectively.
Th ReLUUMMP" (n(")) m E [head, dep], = 1,, |X]
The scores of all possible head-dependen pairs for and all gead-dependenttlabbe triples for label are computed via the Variable-class biaffine classifier [29]]
Rm irrir" ....; ...;. 1,, m E [head, dep]
S* Softmax Reep U1 Rhead + un Rhead + o Rdep + b). k E [are, label).
For arc, U1 E Racy Hnoad U2 E Rucod U3 E R For label, U1 RIPIx Haced, U2 RDDI DDxxHeeed tt3 E D|x where Hhead is the dimension of the head representations, Hdep is the dimension of the dependent representations and the label set. So that Sare E RlXldcu X Inowd and Slabel RI [epp | [K]heed
During training, we aim to optimize the following probability for UD parsing:
|X| Po (o [1, ". P1
Po( = II
where 8 denotes the learnable parameters and yere, y Labet denote the gold-standard head and de- pendency relation for subword 2; subword-level parsing tree Y. The training objective for UD parsing is the cross-entropy. which minimizes the negative log- likelihood::
[X|
CUSL = Σ
log Po (ot + log Po(y Label pii y5 )r
For evaluation, we restore the subword-level score tensors: Sare and glabel to word-level: Šare and Slabel by extracting the first subword of each word. Then, we judge whether the prediction result of Sare is a valid parsing tree. so, we directly extract the corresponding prediction label from Şlabet Otherwise, we use the max spanning tree (MST) algorithm to find the maximum spanning tree based on gare

---

3.4 Universal Structure Integration
In order better integrate linguistic structure knowledge into the output representation of our PrLM, we propose the USI layer, which combines Sure and glabel obtained by the USL layer with H(4) as the final output representations.
We first combine gare and glabel into full label scoring matrix SUS by dot product. gus E R x
xxlepx|X[neee [X [heudX|DD) (Di, in fact, stores the information about the label-head pair probability of each dependent in the sentence. Then we use this label scoring matrix gts as the attention score to obtain a dependent and label specific representation by product summation operation H(4), the result HUS E RIXIx
SUs S glabel, H := Sifk
: H (L) + Oshk.
where Fljx [a + Flas indicates the Einstein summation notation.
Then employ a weight tensor W E R|P)x to aggregate and map the dependent and label specific representation to the final dependency tree-aware representation: HUS E x
H H Wkhm Oink,
HUS GELU(Linear(HH"
where Mine x Flssoo + [ indicates the Einstein summation notation, the dimensions of H R x HX|P), and will be flatten to shape xX before input to Linear.
We also do residual connection to avoid losing the information in H(4,, and we additional Transformer layer to get the final representation.
H (L+1)_ Transformer(H" + +H())
3.5 Pre-training Objectives
We use Masked LM (MLM) as the only pre-training objective other than USL. In MLM, a random sample of the tokens in the input sequence selected and replaced with the special token [MASK]. As described in BERT [M]] 15% of the input tokens are uniformly selected for possible replacement. Of the selected tokens, 80% are replaced with [MASK], 10% are left unchanged, and 10% are replaced by randomly selected vocabulary token. The MLM objective is a cross-entropy loss predicting the masked tokens:
C MLM
Σ (Em(X)
(o (m()))
where m(X) and X (m(x) denote the masked words from X and the rest of words respectively. The training loss of our model calculated by adding the losses of USL and ML...
CUSL + MMLM
3.6 Training Details
Algorithm M shows the training process our model, in which Token denotes the tokenization, Strategy denotes the strategy creating subword-level parsing tree, Opt denotes the optimization strategy and p. Y, w, & denote the learnable parameters Transformer encoder (and token represen- tation), USL layer, USI layer and MI.M decoder respectively. We randomly initialize the model parameters 0. In the first m epochs, USL and ML. are trained as two parallel tasks sharing the parameters in Transformer encoder. That is to say, do not use USI layer to integrate universal structure knowledge for MLM because the parsing capability of the model is too weak in this phase. In the next n steps, we open the USI layer. At this time, the USL objective has converged well, so we reduce the frequency of training USL appropriately according to a certain probability
Experiments
4.1 Setup
Pre-training Data Similar to that of m-BERT, we chose the top 104 languages with the largest Wikipedias and apply exponentially smoothed weighting these languages to balance the Wikipedia
5

---

size of each language for training MLM in the resulted UD-BERT according to our proposed ap- proach. We do not use the NSP objective for UD-BERT. For UD-XLM- ---sLM--Ruse and UD-XLM-R1am Marge we use the same training set from CommonCrawi Corpus as in Conneau et al. [18]- We also used MI.M as the only objective other than USL. for UD-XLMMR
and UID-ILM-RIarge For structure learning, we concatenate all the training TreeBanks covering 60 languages in Universal Dependencies Treebanks (v2.2) [60] as the training set. In addition, we add the sentences in the
training TreeBanks of UD to the training set of MLM letting language modeling directly help our model learn the structure knowledge.
Pre-training Settings In order to be consistent with the baselines and minimize the inequity of the amount of parameters, we use the same settings as the baselines for H, and vocabulary. However, for the number of Transformer layers, since we an additional Transformer layer in USI, the number of layers L prior to this should be reduced by one from the baselines. Specifically, our UD-BERT and UD-XLM-RR Rbase base use Transformer architecture with L = 11, H 768 and A = 12 with a vocabulary of 10k and 250k respectively. Our UD-XLM-R architecture with L 23, H 1024 and A 16 with a 250k vocabulary. We use WordPiece [+II] tokenization of UD-BERT and SentencePiece [62]] tokenization for UD-XLMMR and UD-XLM- Rargee We randomly initialize the model parameters rather than using the pre-trained parameters of m-BERT or XLM-R for fair comparison with the baselines, so as to avoid the improvement of training from more training steps. We train our models with the Adam optimizer [3:3]] using the
uses a large Transformer
parameters: Learning rate = 5c - 5, 31 = 0.9, 32 = 0.98, e = l - 6 and L2 weight decay of 0.01, linear warmup [28]. GELU activation [[=l]] and a dropout rate of 0.1. Models are trained for m = 600, 000 and n = 600, 000 epochs in each phase respectively. with Batch size = 128(sents), and the probability of training USL in the second phase p 0.8. The max sequence length of MLM is 384 and the max sequence length for UD parsing is 256. In the USL layer, we set Hheed 128 and Heep = 64. We list the parameters of our full models and baselines, as well as the data statistics of our training data in Table / in Appendix A.Z.
XNLI: Cross-lingual Natural Language Inference takes two sentences as input and determines
whether one entails the other, contradicts it or neither. XNLI is defined on 15 languages. Each language contains development set with 2,490 sentence pairs and test set with 5,010 sentence pairs. Only English has training data, which is crowd-sourced collection of 433k sentence pairs from MultiNLI [35]. The performance is evaluated by classification accuracy.
XQuAD: Cross-lingual Question Answering Dataset
[6 is benchmark dataset for evaluating
lonsu-linuua question answering performance. The dataset consists of subset of 240 paragraphs and 1,190 question-answw pairs from the development set of SQuAD v1.1 [R] together with their professional translations into 10 languages. The performance is evaluated by F1 and exact match (EM) scores.
Universal Linguistic Structure Parsing: For universal dependency parsing, we evaluate our model on 22 languages in Universal Dependencies Treebanks (v2.7) [30]] whose detail information shown in Table in Appendix A.2. We use the graph-based deep biaffine dependency parsing model [2%]] as our dependency parser. For monolingual evaluation, we train model for each lan- guage on their training set using word, character and POS tag embeddings of dimension 100 and representation from PrLM of dimension 300. For cross-lingual evaluation, we train a single model on English training set using POS tag and representation from PrLM. Unlabeled Attachment Scores (UAS) and Labeled Attachment Scores (LAS) are adopted as the evaluation metries. For universal constituent parsing, we explore the improvement which focuses on parsing nine morphologically rich languages from different language families. We also evaluate our model on PTB and CTB. We use the CRF constituency parsing model [37]] as our constituent parser with word and character embeddings of dimension 100 and representation from PrLM of dimension 300 for monolingual evaluation. For cross-lingual evaluation, we train a single model on PTB using only the representation from PrLM.
our model using SPMRL Shared Task 2014,
4.2 Results and Analysis
To evaluate the multilingual performance and cross-lingual transfer effect of the PrL.M that learns the universal linguistic structure and integrates the universal linguistic structure into the represen-

---

tation explicitly, we conducted experiments on the two typical tasks: universal natural language understanding and the universal linguistic structure parsing. All the scores are average results of five random seeds, indicating that our results are stable.
Universal Natural Language Understanding In Table we show the cross-lingual transfer re- sults (rros--Trrsser of the baselines and our proposed model on the cross-lingual text classifica- tion benchmark XNLI. Meanwhile, we also list the multilingual performance (Train-Trans-FTT, Best-Trans-Eve and All-FT) as a reference. First, compare the results of the source language English. our UD-BERT, UD-XLM-Rpase and UD-XLM- outperform the corresponding m- BERT. XLM-R and XLM-Raagg baselines, demonstrating the universal linguistic structure as pre-training objective and explicitly syntactic structure integration improve the model pre-training and final representations.
Table 1: Results on cross mooss-lngual text classification task, We report the accuracy on each of the 15 XNLI languages and the average accuracy. Results with are from [26]]
Model
em
fr
os de
el
bg ru tr
ar
vi th
zh
hi
Sw
ur
Avg
Train Trans FT: Fine- mubilingad model on each training set translated from English
XIM 2
82.9
77.6
77.9
77.9
77.1
75.7
75.5
72.6
71.2
75.8
73.1
76.2
70.4
66.5
62.4
74.2
Test Trany Eval: Translate test vets to English and use Eigglish only model for evaluation
BERT-en
88.8
81.4
82.3
80.1
80.3
76.2
76.0
75.4
72.0
71.9
75.6
70.0
65.8
65.8
76.2
RoBERTa
91.3
82.9
84,3
81.2
81.7
83,1
78.3
76.8
76,6
74.2
74.1
77.5
70.9
66.7
66.8
77.8
All-FT: Fine-tune madtilingual model on all training sets
XLM (88
84.5
80.1
81.3
79.3
78.6
77.5
75.2
75.6
78.3
75.7
78.3
72.1
69.2
67.7
76.9
XLM [2]
85.0
80.8
81.3
80.3
79.1
80.9
78.3
75.6
77.6
78.5
76.0
79.5
72.9
72.8
68.5
77.8
Unicoder [28]
85.6
81.1
82.3
80.9
79.5
81.4
79.7
76.8
78.2
77.9
77.1
80.5
73.4
73.8
69.6
78.5
XLM-Raa
85.4
81.4
82.2
80,3
80.4
81.3
79.7
78.6
77.3
79.7
77.9
80,2
76.1
73.1
73.0
79.1
XLM- Rage
89.1
85.1
86.6
85.7
85.3
85.9
83.5
83.2
83.1
83.7
81.5
83.7
81.6
78.0
78.1
83.6
Cruss Trunsfer:
tune multilnguul model English training set
XLM [2]
85.0
78.7
78.9
77.8
76.6
77.4
75.3
72.5
73.1
76.1
73.2
76.5
69.6
68.4
67.3
75.1
Unicoder [X)
85.1
79.0
79.4
77.8
77.2
77.2
76.3
72.8
73.5
76.4
73.6
76.2
69.4
69.7
66.7
75.4
M-BERT IMI
82.1
73.8
74.3
71.1
66.4
68.9
69.0
61.6
64.9
69.5
55.8
69.3
60.0
50.4
58.0
66.3
UD-BERT
82.7
74.9
75.2
72.0
67.4
69.2
70.3
62.7
65.8
70.3
59.6
69.7
61.4
51.2
S8.7
67.4
85.8"
779.7
RITT
78.7
777.5
79.6
"8..1
74.
75.8"
"Y6.3"
74.6
7677"
772.3
86.55
68.3"
766.2
UD-XLM-
86.5
80.3
81.6
79.8
78.4
80.0
78,9
75.1
74.4
77.1
75.2
77.3
73.0
67.0
68.8
76.9
XLM-R
89.1
84.1
85.1
83,9
82.9
81.2
79.6
79.8
80.8
78.1 1
80.2
76.9
73.9
73.8
80.9
UD-XLM-
89.4
84.8
85.6
84.5
83.6
84.7
81.2
80.0
81.0
81.9
78.6
80.7
76.8
74.4
74.3
81.4
Second, our UD-BERT and UD-XLM-R performed better in most cases of the 14 transferring target languages. UD-BERT has an average increase of 1.1 when compared to the baseline, UD-XLM- Rhose has an average increase of 0.7, and UD-XLM-R has an average increase of 0.5. This improvement highlights the fact that the cross-lingual transferring ability of our multilingual PrLM has improved as a result of the employment of universal linguistic structures. Furthermore, the cross-lingual transfer effect of our UD-XLM-R outperforms BERT with a similar model struc- ture and parameters, who translated the test set for evaluation. The results of our UD-XLM-R,, R achieved better results than all the methods which leveraging a monolingual language model on the translations, which shows that cross-lingual transfer is more promising mode.
Text classification is relatively simple and intuitive NLU task. To further verify our conclusions, we conducted experiments on a more complex task- cross-lingus Machine Reading Comprehen- sion (MRC). The results on XQuAD dataset are shown in Table E Similarly, we first compare the MRC results on the source language English. The performance of UD-BERT and UD-XLM-R has increased relative to the baseline, which verifies the conclusion that our universal linguistic structure improves the NLU ability of multilingual language model. For the 11 transferring target languages, we observed a similar improvement trend on UD-BERT and UD-XLM-R as in the XNLI task, and the improvement was even greater. The average improvement of UD-BERT, UD-XI DD-MLM-R base and UD-XLM-R large reached 1.0, 0.6, 0.8 Fl scores respectively. Among them, Arabic has the largest improvement, with 1.3, 2.3, and 6.6 F2 scores respectively. All the results here reveal that universal syntactic structure information embedded is effective for cross-lingual MRC task.
Universal Linguistic Structure Parsing Since the universal dependency parsing structure and the dependency parse encoding structure are built into our multilingual PrI.M, to demonstrate that our model learned how to extract linguistic structure and to encode the structure into the output
7

---

Table 2: F2 EM scores XQuAD with English as the source language for each target language.
Cross- Transfer
em
ar
es
ru
th
tr
zh
ro
Avg
Fj
83.5
61.5
70.6
62.6
75.5
59.2
71.3
42.7
55.4
69.5
58.0
72.7
65.2
m-BERT
EM
72.2
45.1
54.0
44.9
56,9
46.0
53.3
33.5
40.1
49.6
48.3
59.9
50.3
Fi
83.9
62.8
72.3
62.9
75.7
59.0
71.6
48.6
55,5
69.9
58.7
73.7
66.2
UD-BERT
EM
72.5
46.3
56.0
45.8
57.3
46.3
54.0
38.5
40.2
49.7
48.6
60.3
51.3
83.6
66.8
74.4
73.0
76.4
68.2
74.3
66.5
68.3
73.7
51.3
77.8
71.2
XLM-Rrus
EM
72.1
49.1
55.7
58.3
51.7
58.1
56.7
52.8
53.8
42.0
62.8
F1
84.0
69.1
74.9
73.5
77.0
68.4
74.3
66.9
69.3
74.1
51.8
78.0
UD-XLM-R
EM
72.5
51.2
60.5
56.0
58.3
57.1
52.9
42.6
63.3
F1 86.5
68.6
80.4
79.8
82.0
76.7
80.1
74.2
75.9
79.1
59.3
83.6
77.2
XI.M-Ringe
EM
75.7
49.0
63.4
61.7
63.9
59.7
64.3
62.8
59.3
59.0
50.0
69.7
61.5
86.8
75.2
80.9
80.0
82.3
77.1
80.3
73.8
76.3
79.5
59.4
83.9
78.0
UD-XLM-R
EM
75.9
58.2
63.8
61.7
64.0
59.8
64.5
62.2
60.5
59.8
49.9
69.9
Table 3: The monolingual UD parsing results (UAS/LAS) on 22 UD Treebanks. means that the
PrLM is tested directly without additional training of a biaffine dependency parsing model.
AI-FFT
m
BERT
UD-BERT
UD-BERT
XLM-Rpuz Ruse LAS
UD-XLM-
Rose
XLM-
KLu-RRuge
UD-XLM MLM-Rurge
UAS
LAS
UAS
LAS
UAS
LAS
UAS
UAS
LAS
UAS
LAS
UAS
bg
94.75
90.88
95.76
91.57
96.12
93.09
96.42
93.56
96.57
93.80
96.53
93.72
96.69
94,04
ca
95.36
93.25
94.97
92.72
95.61
94.27
95.53
94.26
95.78
94.61
95.75
94.58
95,94
94.72
94.38
91.62
95.11
91.94
95.62
93.17
95.60
93.30
95.82
93.43
95.87
93.69
95.99
93.87
nl
94.74
92.72
94.44
91.27
95.38
93.61
95.36
93.44
95.79
93.82
95.63
93.78
96.13
94.50
92.52
91.29
91.34
88.17
93.01
91.43
93.60
91.83
94.15
92.55
93.47
91.73
94.19
92.54
et
90.88
88.95
90.04
86.23
91.65
89.73
92.53
90.78
92.78
91.02
93.16
91.50
93.28
91.68
fi
92.98
90.65
89.41
82.51
94.07
91.89
94.99
93.14
95.15
93.44
95.17
93.46
95.66
94.01
fr
94.12
90.75
95.52
92.20
96.08
94.24
96.10
94.34
96.48
94.69
96.01
94.15
96.53
94.75
90.77
86.83
88.64
81.78
91.08
87.35
91.30
87.42
91.58
87.63
91.39
87.50
91.72
87.94
92.32
89.95
92.52
88.55
93.45
91.06
93.50
91.41
93.87
91.64
93.67
91.48
93.99
91.79
hi 96.54
94.22
95.25
91.34
96.71
94.41
96.73
94,51
97.03
94.99
96.93
94.76
97.17
95.09
id
88.29
83.97
87.73
79.18
87.96
83.72
88.25
84.06
88.51
84.25
88.38
84.27
88.48
84.19
it
95.63
94.01
95.99
93.59
96.32
95.16
96.15
94.93
96.70
95.33
96.26
95.01
96.82
95.60
ko 90.73
88.27
81.90
71.29
90.99
88.67
91.33
89.25
91.42
89,30
92.15
89.79
92.16
89.98
la 85.69
81.84
84.57
79.04
87.01
83.45
86.64
82.99
89.86
86.91
86.97
83.29
90.44
87.33
Iv 91.55
89.06
90.57
85.65
92.06
89.72
93.53
91.25
93.88
91.50
94.23
91,90
94.56
92.52
95.62
93.88
94.61
92.64
96.23
95.17
96.57
95.58
96.71
95.63
96.70
95.64
96.74
95.75
pl 98.15
96.54
96.26
80
98.54
97.14
98.51
97.22
98.80
97.77
98.47
97.08
98.79
97.74
92.70
86.39
93.58
87.38
94.12
89.39
94.25
89.67
94.60
90.11
94.39
89.85
94.73
90.32
ru 95.26
94.00
95.30
92.95
95.74
94.58
96.25
95.26
96.54
95,59
96.38
95.45
96.62
95.72
sk
94.93
91.40
96.22
93.19
96.62
93.95
95.96
93.21
97.49
95.53
95.56
92.74
97.26
95.34
es
94.69
92.89
94.08
90.72
94.92
93.23
94.98
93.44
95.38
93.80
95.33
93.80
95.55
94.09
Avg
93.30
90.61
92.45
87.82
94.06
91.75
94.28
92.04
94.77
92.61
94.47
92.24
94.97
92.89
representations, we performed the universal linguistic structure parsing on UD and multilingual constituent parsing tasks. It is worth noting that, to shield the influence of the increase in the number of parameters caused by the addition of the PrLM to the downstream task model, kept all the PrLM parameters frozen in the universal NLU evaluation.
In Table 3, we evaluated the effect of using the multilingual PrLM to enhance the parsing model on 22 languages of UD dataset respectively, in order analyze how many features useful (i.e., syntactic- aware features) for parsing are provided by the output representation of the various multilingual PrLM. At the same time, to verify that our model has learned syntax, we listed the results that UD-BERT is tested directly 22 languages without additional training of biaffine dependency model.
Comparing UD- UD-BERT and m- m-BERT, our UD-BERT obtained generally better results, except for Indonesian, with a 0.76/1.14 14 UAS/LAS average improvement. The average score of UD-BERT" is comparable to that of m BERT, and it outperforms m-BERT in some languages, indicating that our model has integrated well-performing parser that can be directly applied to practice without additional training. When we further compare UD-BERT to stronger XLM-R and XLM-R baselines, we found that XLM-R achieved similar results as UD-BERT. Since XLM-R uses more data and longer pre-training time, from this point of view, UD-BERT has played role in reducing
8

---

Table 4: Labeled F scores on PTB, CTB, and SPMRL test sets.
AlI-FT
en
ar
eu
fr
de
he
hu
ko
pl
zh
Avg
M-BERT
94.87
88.60
91.06
84,98
90.20
83.35
92.33
89.36
96.36
83.05
90.83
89.54
UD-BERT
95.09
89.97
92.23
85.57
91.06
84.36
93.46
89.72
96.59
85,06
90.99
90.37
XLM-Rauee
95.50
89.67
91.68
85.63
91.54
85.20
93.92
90.82
96.72
86.00
91.84
90.77
UD-XLM-R
95.83
90.22
92.67
86,10
91.98
86.03
94.61
91.32
96.95
86.71
92.30
91.34
XLM-Range
95.74
90.41
92.54
86.47
92.10
86.11
94.56
91.77
97.15
86.80
92.24
91.44
UD-XLM-Rau Rurgge
96.15
90.77
94.13
86.96
92.65
86.97
95.41
92.16
97.33
89.06
92.43
92.18
Table 5: Unlabeled F scores on PTB, CTB, and SPMRL test sets.
Cross-Transfer
em
ar
eu
fr
de
he
hu
ko
pl
SV
zh
Avg
m- m-BERT
95.54
24.22
32.61
59.40
44.36
46.01
57.86
33.99
37.47
71.69
56.87
50,91
UD-BERT
95.93
27.09
35.50
60.87
45.24
46,88
58,99
35.67
39.23
72.20
57.74
52.30
XLM-Rusc
96.19
28.17
34.34
59.70
44.98
47.05
60.26
38.95
38.06
73.68
55.36
52.43
UD-XLM-Ru
96.60
30.46
37.75
61.36
45.52
47.42
62.00
42.94
39.83
73.90
59.97
54.34
XLM-Ruse 96.44
20.36
34.23
59.8I
45.04
47.01
59.37
36.72
37.87
73.43
34,34
31.35
UD-XLM-Rua.
96.73
23.44
37.63
61.18
45.71
47.30
60.46
41.26
38.42
73.81
59.58
53.23
the data and time required for pre-training due to the addition of syntactic supervision information. Further comparing UD-XLM-R with the baseline XLM-R, we found that our method is still helpful on such strong baselines, which improves the parsing performance in each language. In addition, we also evaluate the improvement of our model zero-shot pross-lingual' setting and in low-resource languages. The results are shown in Appendices A.3 and A.4 respectively.
Though verified the linguistic feature extraction and encoding capabilities of our PrLM in the UD parsing, since the evaluation task is the same as in the pre-training, it is hard to illustrate this effect on the cross-task linguistic parsing task. Thus, conduct further exploration constituent parsing benchmarks. Table Hl shows the enhanced ability of the language model for mono- lingual constituent parsing. The comparison demonstrates that with the help of additional universal dependency features in the representations, the constituent parsing performance of UD-BERT and UD-XLM-R has been greatly increased, and the average improvement has reached 0.83, 0.57, and 0.74 respectively. This reflects that the universal syntactic features contained in the representations are very helpful for downstream tasks. In addition, with our syntax- aware multilingual PrLM, the integration of syntactic tree is no longer necessary to change the downstream task model for adding extra syntactic encoders, which will greatly reduce the cost of syntactic tree information application.
multilingual
The multilingual boosting ability of our PrLM is shown in Table AL Similarly, we evaluated the PrLM's cross-lingual transfer ability on constituent parsing in Table 5L and because there are no available universal constituent label annotations, we only report unlabeled F1. Again, our UD- BERT and UD-XLM-R achieve an improvement in all transferring target languages, which shows that the explicit universal features contained in our PrLM can be used to help downstream cross- lingual transfer tasks. Since the universal dependency structure can be derived directly by additional inference in our PrLM's intermediate output, our good tross-lingua transfer ability is interpretable. Furthermore, our studies found that the transfer effect of XLM-Rpase superior to that of XLM- Runge implying that larger model parameters may not always result in higher cross-lingual transfer capabilities.
5 Ablation Study
In our multilingual PrLM, we propose both novel model structure and novel pre-training strategy, whose effectiveness is verified by the ablation study in Table G. We also doubt that the gain is partly derived from the additional UD plain text, although the amount of UD plain text is small. So we also show the results of training the baseline with additional UD plain text (w/ UD plain text) in Table R From the results, if universal dependency parsing is only used as an additional pre-training process (wo US Integration), there is only slight performance improvement compared to the baseline,
In the strict sense, this cannot be totally categorized as zero-shot because the language model part of the full parser has been pre-trained by UD annotations.

---

which shows that just as -re-iiaining approach is not enough to fuse the universal syntax implicitly. When we only employ the model structure without UD pre-training (w/o UD Pre-training), this additional parser structure, even though it is not trained by the UD supervision, can still improve the performance, and the improvement greater than that of w/o US Integration. Comparing m-BERT and m- (-BERT(w/ UD plain tex(), we find that the performance difference between them is very small, and the performance is even lower after the addition of UD plain text. This indicates that a small amount of UD plain text can not directly improve the model. Combining the above results with that our full UD-BERT, we can conclude that the supervised learning the UD structure and its explicit integration into the representation
indispensable options for performance improvement. And our universal structure integration can help the model learn the syntactic knowledge without additional supervision and improve the performance because it has accountable form that can well model the syntactic structure features.
Table 6: Ablation studies on XQuAD and SPMRL.
XQuAD(Cross-Trassfff
SPMRL(AI-FT)
Method
ar
de
en
el
ar
de
en
eu
Fi
EM
Fi
EM
Fi
EM
F1
EM
Fi
Fi
Fi
Fj
m-BERT
61.5
45.1
54.0
83,5
72.2
62.6
44.9
88.6
90.2
94.9
UD plain
61.0
44.6
70.1
53.8
82.9
71.8
62.3
44.8
88.5
90.2
94.6
91.0
UD-BERT
62.8
46.3
72.3
56.0
83,9
72.5
62.9
45.8
90.0
91.1
95.1
92.2
w/o US Integration
61.9
45.4
70.9
54.5
83.8
72.6
62.5
45.0
88.9
90.4
94.8
91.3
w/o UD Pre-training
61.5
45.0
70.7
54.3
83.6
72.2
62.7
45.2
89.1
90.6
94.9
91.5
6 Conclusion
In this work, we propose multilingual PrLM that supports both explicit universal structure learning and implicit language modeling. We chose the universal dependency parses as our universal structure knowledge and evaluate the cross-lingual modeling capability of our model on two rross-lingual NLU tasks and four syntactic parsing datasets. Our model outperforms m- m-BERT and XLM-R in all tasks and achieves state-of-the-art results syntactic parsing. Unlike other works that use the syntax as an objective or feature, our structure learning is used not only for a pre-training objective but also for improving the representation, which makes model both PrLM and universal dependency parser. This will greatly change the way that downstream NLP tasks use syntax, because have explicitly integrated the syntactic knowledge into the representation of the Pr..M.
References
[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers). Minneapolis, Minnesota: Association for Com- putational Linguistics, Jun. 2019, pp. 4171-4186.
[2] Y. Sun, S. Wang, Y. Li, S. Feng, Tian, H. Wu, and H. Wang, "Ernie 2.0: continual pre- training framework for language understanding." in Proceedings of the AAAI Conference Artificial Intelligence, vol. 34, no. 05, 2020, pp. 8968-8977 8975.
[3] I.. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang. J. Gao, M. Zhou, and H.-W. Hon, "Unified language model pre-training for natural language understanding and generation." in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: 120b2za925500aaa17771111555555 Patp Paper,pdf
[4] Y. Liu, M. Ont, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and Stoyanov, "Roberta: robustly optimized BERT pretraining approach," CoRR, vol. abs/1907. 1692, 2019. [Online]. Available: http://arxn.oorraass////...11
10

---

[5] D. Cer, M. Diab, E. Agine, I. Semantic textual similarity multilingual and crosslingual focused evaluation," in Proceedings of the 11th International Workshop Semantic Evaluation (SemFvval-007). Vancouver, Canada: Association for Computational Linguistics, Aug. 2017, pp. 1-14. [Online]. Available:
and L. Specia, "SemEval-2011 task 1:
[6] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ questions for machine comprehension of text," in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguisties, Nov. 2016, pp. 2383-2392. [Online]. Available: http://www.cllwbbbrgaaatttng///
[7] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng. and C. Potts, "Recursive deep models for semantic compositionality over a sentiment treebank," " in Proceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1631-1642.
[8] J. Zhou, Z. Li, and H. Zhao, "Parsing all: Syntax and semantics, dependencies and spans," in Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Associa- tion for Computational Linguistics, Nov. 2020, pp. 4438 4449.
[9] A. Conneau and G. Lample, "Cross-lingual language model pretraining," in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Aché-Buuc E. B. Fox, and R. Garnett, Eds., 2019, pp. 7057-7067. [Online]. Available: httpp::/proccedingg.eerrrr Ahsssrchh
[10] A. Conneau, S. Wu, H. Li, L. Zettlemoyer, and V. Stoyanov, "Emerging cross-lingual structure in pretrained language models," Proceedings of the 58th Annual Meeting of the Asvociation for Computational Linguistics. Online: Association for Computational I. Linguistics, Jul. 2020, Available:
pp. 6022-6034.
[11] W. Wang. Bi, M. Yan, C. Wu, J. Xia, Z. Bao. L. Peng, and L. Si. "Structbert: Incorporating language structures into pre-training for deep language understanding." in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. [Online]. Available:
http:///ppaneeve..eeffonmmm' (ggg//o
[12] J. Zhou, Z. Zhang. H. Zhao, and S. Zhang, "Limit-bert: Linguistics informed multi-task bert," in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
[13] R. Caruana, "Multitask learning: knowledge- based source of inductive bias," Proceedings of the Tenth International Conference on Machine Learning. Morgan Kaufmann, 1993, pp-
41-48.
[14] S. R. Bowman, J. Gauthier, A. Rastogi, R. Gupta, C. D. Manning, and C. Potts, "A fast unified model for parsing and sentence understanding." " in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1466-1477. [Online], Available:
[15] G. Angeli, M. J. J. Premkumar, and C. D. Manning, "Leveraging linguistic structure for open domain information extraction," " in Proceedings the 53rd Annual Meeting the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), 2015, pp. 344-354.
[16] K. Toutanova, X. Lin, W.-t. Yih, H. Poon, and C. Quirk, "Compositional learning of em- beddings for relation paths in knowledge base and text," in Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), 2016, pp. 1434-1444.
[17] R. McDonald, J. Nivre, Y. Quirmbach-Brundagge Y. Goldberg, D. Das, K. Ganchev, K. Hall, S. Petrov, H. Zhang, O. Täckström, C. Bedini, N. Bertomeu Castelló, and J. Lee, "Universal Dependency annotation for multilingual parsing," in Proceedings of the 51 Annual Meeting of the Association for Computational I Linguistics (Volume 2: Short Papers). Sofia, Bulgaria: Association for Computational Linguistics, Aug. 2013, pp. 92-97.
[18] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, Guzmán, Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, "Unsupervised cross-lingual representation learning
11

---

at scale," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440-8451.
[Online]. Available:
[19] D. Seddah, S. Kübler, and R. Tsarfaty, "Introdueing the SPMRL 2014 shared task on parsing morphologically-rich languages." in Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non- Con-Canonical Languages. Dublin, Ireland: Dublin City University, Aug. 2014, pp. 103-109. [Online].
Available:
14-6111
[20] M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz, "Building a large annotated corpus of English: The Penn Treebank," Computational Linguistics. vol. 19, no. 2, pp. 313-330, 1993. [Online]. Available:
[21] N. Xue. F.-D. Chiou, and M. Palmer, "Building large-scale annotated Chinese corpus." in COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
[22] W. Wang. B. Bi, M. Yan, C. Wu, J. Xia, Z. Bao, L. Peng, and L. Si. "Structbert: Incorporating language structures into pre-training for deep language understanding." in International Conference on Learning Representations, 2020. [Online]. Available: https//openneviewnne//mmm (igg:/ooummndddBBB(IISSS
[23] X. Ren, X. Sun, H. Wang, and Q. Liu, "Towards semantics-enhanced pre-training: Can lexicon definitions help learning sentence meanings?" 2021.
[24] E. Strubell, P. Verga, D. Andor, D. Weiss, and A. McCallum, "Linguisticall--inforeed self-attention for semantic role labeling," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 5027-5038. [Online]. Available: https///ccannhhlooy..r///// Drg//D8--/DD1
[25] D. Sachan, Y. Zhang, P. Qi, and W. L. Hamilton, "Do syntax trees help pre-trained transformers extract information?"
in Proceedings of the 16th Conference of the European Chupter of the Association for Computational Linguistics: Main Volume. Online: Association for Computational Linguistics, Apr. 2021, pp. 2647-2661. [Online]. Available: https://aclantoolgy.ooo 2021.e2cl-maaamieee.ee
[26] H. Huang, Y. Liang, N. Duan, M. Gong, L. Shou, D. Jiang, and M. Zhou, "Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks," in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)), Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 2485-24552442 2494. [Online]. Available: http://www...llweoorrrrtttooo///
[27] W. Ahmad, H. Li, K.-W. Chang, and Y. Mehdad, "Syntax-augmenn for cross-lingual transfer," in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). Online: Association for Computational Linguisties, Aug. 2021, pp. 4538-4554. [Online]. Available:
multilingual BERT
[28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https: C4455333314444444 b Paper.pdf
/proceedings.neumm http:///nssnnnuussssscppppeppeee
[29] T. Dozat and C. D. Manning, "Deep biaffine attention for neural dependency parsing," ICLR, vol. abs/161 1. 1.01734. 2017.
[30] J. Nivre, M.-C. de Marneffe, F. Ginter, J. Hajič, C. D. Manning, S. Pyysalo, S. Schuster, F. Tyers, and D. Zeman, "Universal Dependencies v2: An evergrowing multilingual treebank collection." in Proceedings of the 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 4043-4043. [Online]. Available: http:://www..eewebooogonnhhho//p
[31] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa,
12

---

A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, "Google's ' neural machine translation system: Bridging the gap between human and machine translation," CoRR, vol. abs/1609.08 (06.999911000 010000884444 144, 2016. [Online]. Available: http://axiiv.org/bb///oo 1609.08144
[32] T. Kudo and J. Richardson, "Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, E. Blanco and W. Lu, Eds. Association for Computational Linguistics, 2018, pp. 66-71. [Online]. Available:
htpss//to.orr/11118633/////1
[33] D. Kingma and Ba, "Adam: A method for stochastic optimization," in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Bengio and Y. LeCun, Eds., 2015. [Online]. Available:
http://rxiv.org/abb// (11.g//////11..988..
[34] D. Hendrycks and K. Gimpel, "Bridging nonlinearities and stochastic regularizers with gaussian error linear units," CoRR, vol. abs/1606.08415. 2016. [Online]. Available:
http://axxv.orr/ab///bb...8
[35] A. Williams, N. Nangia, and Bowman, "A broad coverage challenge corpus for sentence understanding through inference," in Proceedings of the 2018 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume I (Long Pupers). New Orleans, Louisiana: Association for Computational Linguistics. Jun. 2018, pp. 1112-1122. [Online]. Available:
[36] M. Artetxe, S. Ruder, and D. Yogatama. "On the cross-lingual transferability of monolingual representations," Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 4623 4637. [Online]. Available: http://www.aawwbnaaaattyy///aaaa
[37] Y. Zhang, H. Zhou, and Z. Li, "Fast and accurate neural erf constituency parsing," in Proceedings of the TwentyNinth International Joint Conference on Artificial Intelligence, IJCAI-20, C. Bessiere, Ed. International Joint Conferences on Artificial Intelligence Organization, 7 2020, pp. 4046-4053, main track. [Online]. Available: https://dos.org//u) 20025/1)
13


Reviews:
[{"review_id": "s7uCtnw60Pu", "reviewer": "Reviewer_kc6w", "summary": "\nThis paper proposes to include syntactic supervision from the Universal Dependency (UD) dataset for pre-training Transformer-based multilingual encoders, in addition to the typical masked language modeling objective.\n\nThe proposed method not only utilizes the parsing objective to guide the training of a portion of the model parameters but also incorporates the features computed for parsing in the process of deriving the final contextualized word representations.\n\nThe resulting multilingual pre-trained language models are evaluated on cross-lingual NLI & QA in addition to mono/cross-lingual constituency/dependency parsing. \nThe paper claims that the newly introduced pre-training strategy is superior to that of the existing baselines in most cases.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n### Strengths\n- According to the paper, the proposed method shows its effectiveness in diverse downstream tasks such as cross-lingual NLI & QA and mono/cross-lingual constituency/dependency parsing.\n- The proposed approach attempts to directly leverage the features derived from the parsing objective to generate the final representations that can consider syntactic aspects of input sentences.\n\n\n### Weaknesses\n- A more clear and accurate method specification is needed (Section 3.3): It is hard to understand the provided equations including one containing softmax. \nFor instance, in the softmax function, the first term $R^T_{dep}U_1R_{head}$ should be a matrix while the second and third terms should be vectors. I don't understand how you can directly add that matrix and those vectors without extra processing. \nFurthermore, it is unclear in the paper how the terms $R_m$ and $S^k$ and the final universal structure learning loss ($L_{USL}$) can be connected.\nPlease correct this if I'm wrong. \nIt seems that the formulations are too simplified to clearly state their meaning.\n- An existence of confounding factors that can have an impact on performance: To the best of my knowledge, the proposed method appends additional parsing-specific and final integration modules on top of the typical Transformer architecture which is widely used for pre-training. This implies that the models proposed by this work naturally have more parameters than existing baselines such as m-BERT and XLM-R (therefore, please report the number of parameters of each approach).\nFurthermore, the proposed models can see more **plain** sentences from the UD dataset in addition to getting access to structural supervision from parse trees.\nConsidering the aforementioned confounding factors, I'm not sure the increase in performance is entirely due to the effectiveness of structural supervision.\nAlthough the ablation study tried to mitigate some related concerns, the claim of the paper would be much more persuasive if additional experiments can be conducted in a more rigorous condition, e.g., adjusting the number of parameters of each model to a comparable extent.\n\n\n### Questions to the authors\n- In the dependency parsing experiment, the authors decided to train a separate parser instead of using one obtained during pre-training. I'm wondering it's because the pre-trained model cannot be utilized as an effective parser or there exist another reasons in the experiment setup. If we should train a separate parser for performance even in the case where we already included our training dataset in the pre-training phase, I'm not sure the proposed method is attractive especially in terms of data efficiency.\n- Can you imagine some tasks (other than ones presented in the paper) that can be much more benefitted from explicitly considering syntactic information? The experimental results reported in the paper seem to be a little bit incremental, and therefore, if you can find such a proper application, I believe that the contribution of this paper would be much more valuable.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nPlease refer to the Weaknesses part in the main review.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Oz1mIC6AqfA", "reviewer": "Reviewer_JRfY", "summary": "This paper presents a multilingual joint language model and syntactic\ndependency parser. The model consists of a multilingual transformer\nmodel (BERT or XLM) with a deep biaffine parser (Dozat & Manning,\n2017). The outputs of each are then combined to form a final\ntoken-level output layer. Although prior work has already explored the\nincorporation of additional structution into a language model\n(e.g. StructBERT and LIMIT-BERT), this paper differs by applying its\nnew model to the multilingual transfer setting and by incorporating\nthe scores of the syntactic parser back into the embedding\nrepresentation. Training mirrors prior work by jointly optimizing a LM\nobjective and a parsing objective -- the latter on supervised\ndata. Experiments on XNLI, XQuAD, UD dependency and consituency\nparsing, show that the new model achieves improvements over the non-structured baseline transformers.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Overall the paper is clearly written and offers two extensions beyond prior work:\n  1. The output scores of the parser are combined with the language model to produce the final token embeddings. This is achieved by parsing at the token level (rather than the word level as in LIMIT-BERT).\n  2. Applying the model to a multilingual setting. \n- Although these novelty of the technical contributions are not very grand, the results are very compelling. Improvements are achived on XNLI, XQuAD, and UD dependency/constituency parsing.\n\nWeaknesses:\n- The primary ommission is any empirical comparison with the most similar prior work: StructBERT and LIMIT-BERT. Like the UD-BERT/XLM model proposed in this paper, these prior efforts to incorporate linguistic information in pre-training may have implicitly learned useful linguistic structure information within the shared parameters (even though the sharing is less explicit than in this paper). As such, there remains an open question: does feeding the structure back in as done in this paper outperform the approaches taken in StructBERT and LIMIT-BERT? Although these models were only applied to English, this model could straightforwardly be applied in the same setting. (Note that while the ablation experiments in this paper suggest that this feeding back approach is effective, it might be that there are other aspects of those models that provide some benefit not realized in the present model.)\n- Other differences between UD-BERT/XLM (this paper) and Struct-BERT and LIMIT-BERT are not addressed. For example, is parsing at the token level (as in this paper) advantageous over the word-level approach of LIMIT-BERT? Is the sentence structural objective of Struct-BERT complimentary to the parsing objective in this paper?\n- One of the main claims of this paper is that linguistic structure information is encoded by the model. However, no effort is made to analyze how the linguistic information affects the output embeddings. What does the summation approach taken by this paper accomplish? Were any alternatives considered for how to linearize the dependency scores?\n- The paper does not address the impact on languages not included in the supervised parsing data. The model trains the parser submodel on 60 UD languages and tests on 22 UD languages. The language model is trained on 104 languages from Wikipedia. How does the model perform on languages that were *not* included in the original 60 UD training languages? For example, does performance improve or degrade on those languages which are truly low-resource?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Mostly, but the paper makes no mention of the (possible) limititations of the model with regard to languages that did not appear in the original supervised parsing data.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.75"}, "extra_text": {}}, {"review_id": "L2y5ObEUAxf", "reviewer": "Reviewer_uazG", "summary": "The paper proposes to pre-train multilingual encoders explicitly on dependency parsing and language modeling to improve cross-lingual transfer learning. The proposed method, UD-BERT composed of the Transformer encoder along with a universal structure learning layer and a universal structure integration layer. The experiment results show the promise of the proposed method. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThe proposed idea is not new, however, it probably is not investigated for multilingual representation learning.\n\n**Quality and Clarity**\n\nThe work has a balance between modeling and experiments. The writing is clear; all the model components, experiment settings, experiment tasks, and results are sufficiently explained.\n\n**Significance**\n\nThis paper investigates the advantage of using the universal linguistic structure to improve cross-lingual transfer. The proposed technique has a value, the empirical results show improvements over the baselines.\n\n**Weaknesses**\n\nI see one weakness in the experiments which forced me to give a lower score to this paper. The evaluation results show a very marginal improvement over the baseline. I am worried this could be due to variances across experiments. It seems like the authors didn't perform the experiments with different random seeds. For multilingual experiments, it is highly recommended to run experiments multiple times and report the average scores.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any discussion, perhaps it is not required.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JEDcVQEC5ne", "reviewer": "Reviewer_ZFQx", "summary": "This paper proposes to pre-train multilingual language models with the knowledge of dependency parsing. The universal structure knowledge is used in two ways, the learning objective and improvement for representation learning for mask language model. The effectiveness of proposed approach is verified for two pre-trained models, mBERT and XLM on  several downstream tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the idea of leveraging syntactic structure knowledge is not new, this paper proposes a reasonable and interesting way to achieve that goal – multi-task learning with explicit learning objective and improvement for representation learning by leveraging the hidden states of dependency parsing. \n\nThe authors verify its effectiveness on XNLI and XQuAD based on BERT and XLM-R, and it also shows improvement on the linguistic structure parsing task. \n\nOverall speaking, the UD-aware approach for multilingual pre-training looks interesting and experimental results are convincing. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to multilingual pretraining that (a) incorporates supervised dependency parsing as an auxiliary objective and (b) incorporates dependency scores back into the encoder itself. Results on XNLI, XQuAD, and UD dependency and constituency parsing show gains over baselines that do not use syntactic structure. Reviewers are split with two in favor of rejection and two voting for acceptance. Most reviewers viewed both the proposed approach and the positive results as potentially impactful. However, a serious concern was surfaced. Specifically, that the experimental comparisons conflate the two potential contributions: (1) gains from incorporating parsing scores back into the encoder, and (2) gains from training in a multilingual setting. These points need to be separately evaluated in the context of missing, but closely related baselines. For example, how does the method compare with related baselines like Struct-BERT or LIMIT-BERT that also consider syntactic structure, but in a monolingual setting? Does incorporating parsing scores back into these encoders increase performance in a monolingual setting? Do these baselines improve in a multi-lingual setting without other architectural changes? These concerns sway me to recommend rejection for the current draft. But I strongly encourage authors to resubmit with additional experiments.
Decision: Accept (Poster)