Multilingual Pre-training with Universal Dependency Learning
Authors: Kailai Sun
Zuchao Li
hai zhao
Venue: NeurIPS 2021 Poster (2021)
Keywords: natural language processing
multilingual pre-trained language model
universal syntactic dependency parsing

Abstract: The pre-trained language model (PrLM) demonstrates domination in downstream natural language processing tasks, in which multilingual PrLM takes advantage of language universality to alleviate the issue of limited resources for low-resource languages. Despite its successes, the performance of multilingual PrLM is still unsatisfactory, when multilingual PrLMs only focus on plain text and ignore obvious universal linguistic structure clues. Existing PrLMs have shown that monolingual linguistic structure knowledge may bring about better performance. Thus we propose a novel multilingual PrLM that supports both explicit universal dependency parsing and implicit language modeling. Syntax in terms of universal dependency parse serves as not only pre-training objective but also learned representation in our model, which brings unprecedented PrLM interpretability and convenience in downstream task use. Our model outperforms two popular multilingual PrLM, multilingual-BERT and XLM-R, on cross-lingual natural language understanding (NLU) benchmarks and linguistic structure parsing datasets, demonstrating the effectiveness and stronger cross-lingual modeling capabilities of our approach.

Multilingual Pre-training with Universal Dependency Learning
Kailai Sun", Zuchao Li", Hai Zhao"
DDpaatteen of Computer Science and Engineering, Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University (kaishu2. O, charlee) @u. edu. en, zhaohai@cs sj tu. edu. cn
Abstract
The pre-trained language model (PrLM) demonstrates domination in downstream natural language processing tasks, in which multilingual PrLM takes advantage language universality to alleviate the issue of limited resources for low-resource languages. Despite its successes, the performance of multilingual PrLM is still un- satisfactory, when multilingual PrLMs only focus plain text and ignore obvious universal linguistic structure clues. Existing Pr..Ms have shown that monolingual linguistic structure knowledge may bring about better performance. Thus pro- pose novel multilingual PrLM that supports both explicit universal dependency parsing and implicit language modeling. Syntax in terms of universal dependency parse serves as not only pre-training objective but also learned representation in our model, which brings unprecedented PrLM interpretability and convenience in downstream task use. Our model outperforms two popular multilingual PrLM, multilingual-BERR and XLM-R, on cross-lingual natural language understanding (NLU) benchmarks and linguistic structure parsing datasets, demonstrating the effectiveness and stronger cross-lingual modeling capabilities our approach.
Introduction
The pre- pre-trained language model (PrLM) such as BERT [0]] and many kinds of its variants [2] B. [] have proved their effectiveness in many downstream natural language processing (NLP) tasks in-
cluding semantic textual similarity [B]], question answering [[6], sentiment classification [[Z], linguis- tie structure [4, 8] and so on. Most of these PrLM are aimed at languages that with a large amount of available linguistic resources and widely used, such English. However, it is not realistic to train an individual PrLM for all languages, especially for those low-resource languages. As a result, several multilingual PrLMs which take advantage of language universality have been published and shown good cross-lingual performance on several NLP tasks.
Despite its successes, the unsupervised method typically used by multilingual PrLMs makes cross- lingual transfer inefficiency and keeps the learning still challenging. Improvement can be made by adding explicit cross-lingual signals including bitext (XLM) [X] and word translation pairs from dictionary [IU]. This suggests that the effectiveness of multilingual PrLM can be further improved by integrating explicit universal linguistic characteristics. Existing PrLMS (III [2] have tried to incorporate monolingual linguistic structure knowledge to improve the performance across multi-
ple linguistics tasks by Multi-Task Learning (MTL) [K]]] However, the combination of universal
"oorrespnddigg author: These authors made equal contribution. This work supported by Key Projects National Natural Science Foundation China (U1836222 and 6173301 Project from Chinese National Key Laboratory of Science and Technology on Information System Security.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

linguistic structure knowledge has not been explored in the multilingual area. Learning universal knowledge across languages is more complex than learning monolingual knowledge, so a better integrating method than MTL needs to be explored.
Syntactic dependency parsing disclosing syntactic relations between words in sentence, has been found to be extremely useful for many NLP tasks [IEL, 16, [[]]. The syntactic dependency parsing also limited by low-resource languages. To meet the huge demand for training syntactic parser among various languages, the project of universal dependencies (UD) Treebanks was launched [[Z]] which provides a uniform syntactic parsing structure for different languages. Therefore, UD offers an excellent universal structure characteristic, which is worth exploiting for the multilingual PrLM.
In this paper, propose multilingual PrLM that supports both explicit universal dependency parsing and implicit language modeling. Unlike using MTI. in monolingual works, we embed a parsing scorer in our PrI.M, and directly optimizes this scorer and the encoders below it with UD) pre-training: meantime, we propose a structural encoder to encode the predicted structure given by the parsing scorer and integrated it into the final representation for other pre-training or downstream training process. Our approach can be smoothly applied to a variety of multilingual PrLM such as Multilingual-BER (m-BERT) [M]] and XLM-R [[S].
To verify the cross- pooss-lingual modeling capabilities of our model, we carry on the experiments on both cross-lingual NLU benchmarks: XNLI and XQuAD, and linguistic structure parsing datasets: UD3 v2.7, SPMRL' 14 [19), English Penn Treebank (PTB) 3.0 [20] and the Chinese Penn Treebank (CTB) 5.1 [2I]]. Our empirical results show that universal structure knowledge learnt and integrated can indeed help the multilingual PrLM obtain better universal linguistic word representations and outperform m-BERT and XLM-R baselines in all the above tasks.
2 Related Work
Monolingual PrLM with structure learning Previous works have tried to improve monolingual
[5] Some of them use plain text to implicitly learn structural knowledge, such as StrucBERT [2). which lets the PrLM encode the dependency between sentences and words by adding the word-level ordering and sentence-level ordering objec- tives during pre-training. However, this may not be suitable for multilingual setting. Since there are obvious differences in the word order of different languages, putting disordered plain text of multi- ple languages together for common training will lead to confusion in implicitly learning structural information. Contrastively, we use annotated syntactic structures knowledge UD that provide clear guidance for improving cross-lingual representation in multilingual PrI.M.
PrLM by learning linguistic structure [4, IZ 22
25
24
Other works use annotated syntactic knowledge as the structural knowledge such LIMIT-BERT [122] and LISA [24]] LIMIT-BERT is a monolingual PrL.M that achieves advancement in several parsing datasets and NLU tasks by performing MTL on multiple linguistic tasks including syntactic parsing. The main difference between LIMIT-BERT and our method is that we do not regard parsing as a pre-training objective, but use the parsing component as an intermediate structure to explicitly learn and extract of the syntactic structure, thereby reducing the black box characteristics of the model. LISA Transformer model that uses syntax information enhance SRL. It incorporates syntactic information by training one attention head predicting syntactic dependency arc. First of all, LISA only aims one specific task, so doubtful whether its method can be applied to the PrL. and improve the performance of multiple tasks. Secondly, LISA can only use the are feature of syntax but not the relations, while our approach of syntactic structural integration can integrate both of the are and relation features learned by our model into the final universal representation. By the way, our PrLM can be used directly as a well-behaved parser, but LISA clearly does not.
Multilingual PrLM with explicit cross-lingual signals Cross-lingual pre-training tasks includ- ing cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model can take advantage of bitext to learn the mappings among different languages from more perspectives [26]]. Recently, Ahmad et al. [27] proposes a multilingual PrLM training mBERT using an auxiliary objective to encode the universal dependency tree structure that helps cross-lingual transfer. However, our work significantly differs from theirs. We regard parsing not only as pre-training objective, but also an active part of the overall PrLM structure, so this part

---

US Learning Layer
L
Multi-layer Transformers
S
US Integration Layer
Token Embedding
Position Embedding
[CLS]
[SEP]
A
[CLS] X [MASK] X
[SEP]
Masked LM
M.M
Figure I: The model architecture for UD-PrLM.
of the structure will further exert its effectiveness in downstream tasks rather than discarding it in the fine finettuning stage, which increases the adaptability from pre-training fine-tuning phase.
Universal Dependency as Language Modeling
In this work, we chose UD parse as our universal structure knowledge. Our model includes five modules: token representation, multi-layer Transformers, universal structure learning (USL) layer, universal structure integration (USI) layer and pre-training objectives. Figure shows the full model architecture of our method.
3.1 Token Representation
Take BERT [I]] as an example, in the token representation layer, given input sentence X, the sentence concatenated with two special tokens "[CLS]" and "[SEP]": [CLS], 22 .., TN, [SEP], and [CLS] is also used as the dummy ROOT node in UD training process. The input X is mapped into sequence of input embedding vectors [e1, e2...., Exxil, one for each token, which is sum of the corresponding word and positional embeddings.
Since the UD parse tree is annotated at word-level and the input sequence X in PrLM is based on subword tokenization, in order to sufficiently improve the representation of tokens and adapt to the subword-level tasks, we propose conversion strategy that extends the parsing tree from the word level: Y to the subword level: Y. and use the subword-level parsing tree as the training objective for USL. Detailed strategy description is shown in Appendix [A.I.
3.2 Multi-layer Transformers
The multi-layer Transformers architecture in model is adapted from Vaswani et al. a2.] which transforms the input embedding vectors into a sequence contextualized representation vectors H [h,, .h,,... hxi]] shared across different tasks. We Transformer architecture with L layers, A self-attention heads for each block and hidden size H:
H L) = Transformers(Embl (mb + PosEnccdingg(X)

---

Algorithm 1: Training Process
Input: MLM training data XMLM, UD Treebanks (XuD, YUD), Parameters: (p, 7, w, 0),
Probability of training USL: p.
D XUD. YUD <- Token ToMMn U XUD), Token(XUD), Strategy Token)
2 Initialize 8g randomly;
for t <- to do
L 0+t1 < C + nr XtD; YUD) + LMLM (pr di, DM
5 for t < m to m do
if rrrandom.nifomm
then
O++1 < Opt(O,, CUSL(pp + nr XiD YUD) + CULM + + + de, D DMLM))
else
B.t1 t Opt
M.M (po + 7 + we + Φ1. D DMaM));;
Output: On.
3.3 Universal Structure Learning
Our USL layer follows the state-of-the-arr graph-based deep biaffine dependency parser [22]. We replace the BiLSTM encoder with the multi-layer Transformers architecture and use the hidden state of its last layer as the output of encoder H (.) = [11"), (L) _n ,..,,,[[]]]
(L))
For both are and label predictions, two separate MLPs are used to distinguish two kinds of low- dimensional vectors as head and dependent representations respectively.
Th ReLUUMMP" (n(")) m E [head, dep], = 1,, |X]
The scores of all possible head-dependen pairs for and all gead-dependenttlabbe triples for label are computed via the Variable-class biaffine classifier [29]]
Rm iffin" ....; ...;. 1,, m E [head, dep]
S* Softmax Reep U1 Rhead + un Rhead + o Rdep + b). k E [are, label).
For arc, U1 E Racy Hnoad U2 E Rucod U3 E R For label, U1 RIPIx Haced, U2 RDDI DDxxHeeed tt3 E D|x where Hhead is the dimension of the head representations, Hdep is the dimension of the dependent representations and the label set. So that Sare E RlXldcu X Inowd and Slabel RI [epp | [K]heed
During training, we aim to optimize the following probability for UD parsing:
|X| Po (o [1, ". P1
Po( = II
where 8 denotes the learnable parameters and yere, y Labet denote the gold-standard head and de- pendency relation for subword 2; subword-level parsing tree Y. The training objective for UD parsing is the cross-entropy. which minimizes the negative log- likelihood::
[X|
CUSL = Σ
log Po (ot + log Po(y Label pii y5 )r
For evaluation, we restore the subword-level score tensors: Sare and glabel to word-level: Šare and Slabel by extracting the first subword of each word. Then, we judge whether the prediction result of Sare is a valid parsing tree. so, we directly extract the corresponding prediction label from Şlabet Otherwise, we use the max spanning tree (MST) algorithm to find the maximum spanning tree based on gare

---

3.4 Universal Structure Integration
In order better integrate linguistic structure knowledge into the output representation of our PrLM, we propose the USI layer, which combines Sure and glabel obtained by the USL layer with H(4) as the final output representations.
We first combine gare and glabel into full label scoring matrix SUS by dot product. gus E R x| XXXX [heudX|DD) (Di, in fact, stores the information about the label-head pair probability of each dependent in the sentence. Then we use this label scoring matrix gts as the attention score to obtain a dependent and label specific representation by product summation operation H(4), the result HUS E RIXIx
SUs S glabel, H := Sifk
: H (L) + Oshk.
where Fljx [a + Flas indicates the Einstein summation notation.
Then employ a weight tensor W E R|P)x to aggregate and map the dependent and label specific representation to the final dependency tree-aware representation: HUS E x
H H Wkhm Oink,
HUS GELU(Linear(HH"
where Mine x Flssoo + [ indicates the Einstein summation notation, the dimensions of H R x HX|P), and will be flatten to shape xX before input to Linear.
We also do residual connection to avoid losing the information in H(4,, and we additional Transformer layer to get the final representation.
H (L+1)_ Transformer(H" + +H())
3.5 Pre-training Objectives
We use Masked LM (MLM) as the only pre-training objective other than USL. In MLM, a random sample of the tokens in the input sequence selected and replaced with the special token [MASK]. As described in BERT [M]] 15% of the input tokens are uniformly selected for possible replacement. Of the selected tokens, 80% are replaced with [MASK], 10% are left unchanged, and 10% are replaced by randomly selected vocabulary token. The MLM objective is a cross-entropy loss predicting the masked tokens:
C MLM
Σ (Em(X)
(o (mx)))
where m(X) and X (m(x) denote the masked words from X and the rest of words respectively. The training loss of our model calculated by adding the losses of USL and ML..
CUSL + MMLM
3.6 Training Details
Algorithm M shows the training process our model, in which Token denotes the tokenization, Strategy denotes the strategy creating subword-level parsing tree, Opt denotes the optimization strategy and p. Y, w, & denote the learnable parameters Transformer encoder (and token represen- tation), USL layer, USI layer and MI.M decoder respectively. We randomly initialize the model parameters 0. In the first m epochs, USL and ML. are trained as two parallel tasks sharing the parameters in Transformer encoder. That is to say, do not use USI layer to integrate universal structure knowledge for MLM because the parsing capability of the model is too weak in this phase. In the next n steps, we open the USI layer. At this time, the USL objective has converged well, so we reduce the frequency of training USL appropriately according to a certain probability
Experiments
4.1 Setup
Pre-training Data Similar to that of m-BERT, we chose the top 104 languages with the largest Wikipedias and apply exponentially smoothed weighting these languages to balance the Wikipedia
5

---

size of each language for training MLM in the resulted UD-BERT according to our proposed ap- proach. We do not use the NSP objective for UD-BERT. For UD-XLM- ---sLM--Ruse and UD-XLM-R1am Marge we use the same training set from CommonCrawi Corpus as in Conneau et al. [18]- We also used MI.M as the only objective other than USL. for UD-XLMMR
and UID-ILM-RIarge For structure learning, we concatenate all the training TreeBanks covering 60 languages in Universal Dependencies Treebanks (v2.2) [60] as the training set. In addition, we add the sentences in the
training TreeBanks of UD to the training set of MLM letting language modeling directly help our model learn the structure knowledge.
Pre-training Settings In order to be consistent with the baselines and minimize the inequity of the amount of parameters, we use the same settings as the baselines for H, and vocabulary. However, for the number of Transformer layers, since we an additional Transformer layer in USI, the number of layers L prior to this should be reduced by one from the baselines. Specifically, our UD-BERT and UD-XLM-RR Rbase base use Transformer architecture with L = 11, H 768 and A = 12 with a vocabulary of 10k and 250k respectively. Our UD-XLM-R architecture with L 23, H 1024 and A 16 with a 250k vocabulary. We use WordPiece [+II] tokenization of UD-BERT and SentencePiece [62]] tokenization for UD-XLMMR and UD-XLM- Rargee We randomly initialize the model parameters rather than using the pre-trained parameters of m-BERT or XLM-R for fair comparison with the baselines, so as to avoid the improvement of training from more training steps. We train our models with the Adam optimizer [3:3]] using the
uses a large Transformer
parameters: Learning rate = 5c - 5, 31 = 0.9, 32 = 0.98, e = l - 6 and L2 weight decay of 0.01, linear warmup [28]. GELU activation [[=l]] and a dropout rate of 0.1. Models are trained for m = 600, 000 and n = 600, 000 epochs in each phase respectively. with Batch size = 128(sents), and the probability of training USL in the second phase p 0.8. The max sequence length of MLM is 384 and the max sequence length for UD parsing is 256. In the USL layer, we set Hheed 128 and Heep = 64. We list the parameters of our full models and baselines, as well as the data statistics of our training data in Table / in Appendix A.Z.
XNLI: Cross-lingual Natural Language Inference takes two sentences as input and determines
whether one entails the other, contradicts it or neither. XNLI is defined on 15 languages. Each language contains development set with 2,490 sentence pairs and test set with 5,010 sentence pairs. Only English has training data, which is crowd-sourced collection of 433k sentence pairs from MultiNLI [35]. The performance is evaluated by classification accuracy.
XQuAD: Cross-lingual Question Answering Dataset
[6 is benchmark dataset for evaluating
lonsu-linuua question answering performance. The dataset consists of subset of 240 paragraphs and 1,190 question-answw pairs from the development set of SQuAD v1.1 [R] together with their professional translations into 10 languages. The performance is evaluated by F1 and exact match (EM) scores.
Universal Linguistic Structure Parsing: For universal dependency parsing, we evaluate our model on 22 languages in Universal Dependencies Treebanks (v2.7) [30]] whose detail information shown in Table in Appendix A.2. We use the graph-based deep biaffine dependency parsing model [2%]] as our dependency parser. For monolingual evaluation, we train model for each lan- guage on their training set using word, character and POS tag embeddings of dimension 100 and representation from PrLM of dimension 300. For cross-lingual evaluation, we train a single model on English training set using POS tag and representation from PrLM. Unlabeled Attachment Scores (UAS) and Labeled Attachment Scores (LAS) are adopted as the evaluation metries. For universal constituent parsing, we explore the improvement which focuses on parsing nine morphologically rich languages from different language families. We also evaluate our model on PTB and CTB. We use the CRF constituency parsing model [37]] as our constituent parser with word and character embeddings of dimension 100 and representation from PrLM of dimension 300 for monolingual evaluation. For cross-lingual evaluation, we train a single model on PTB using only the representation from PrLM.
our model using SPMRL Shared Task 2014,
4.2 Results and Analysis
To evaluate the multilingual performance and cross-lingual transfer effect of the PrL.M that learns the universal linguistic structure and integrates the universal linguistic structure into the represen-

---

tation explicitly, we conducted experiments on the two typical tasks: universal natural language understanding and the universal linguistic structure parsing. All the scores are average results of five random seeds, indicating that our results are stable.
Universal Natural Language Understanding In Table we show the cross-lingual transfer re- sults (rros--Trrsser of the baselines and our proposed model on the cross-lingual text classifica- tion benchmark XNLI. Meanwhile, we also list the multilingual performance (Train-Trans-FTT, Best-Trans-Eve and All-FT) as a reference. First, compare the results of the source language English. our UD-BERT, UD-XLM-Rpase and UD-XLM- outperform the corresponding m- BERT. XLM-R and XLM-Raagg baselines, demonstrating the universal linguistic structure as pre-training objective and explicitly syntactic structure integration improve the model pre-training and final representations.
Table 1: Results on cross mooss-ingual text classification task, We report the accuracy on each of the 15 XNLI languages and the average accuracy. Results with are from [26]]
Model
em
fr
os de
el
bg ru tr
ar
vi th
zh
hi
Sw
ur
Avg
Train Trans FT: Fine- mubilingad model on each training set translated from English
XIM 2
82.9
77.6
77.9
77.9
77.1
75.7
75.5
72.6
71.2
75.8
73.1
76.2
70.4
66.5
62.4
74.2
Test Trany Eval: Translate test vets to English and use Eigglish only model for evaluation
BERT-en
88.8
81.4
82.3
80.1
80.3
76.2
76.0
75.4
72.0
71.9
75.6
70.0
65.8
65.8
76.2
RoBERTa
91.3
82.9
84,3
81.2
81.7
83,1
78.3
76.8
76,6
74.2
74.1
77.5
70.9
66.7
66.8
77.8
All-FT: Fine-tune madtilingual model on all training sets
XLM (88
84.5
80.1
81.3
79.3
78.6
77.5
75.2
75.6
78.3
75.7
78.3
72.1
69.2
67.7
76.9
XLM [2]
85.0
80.8
81.3
80.3
79.1
80.9
78.3
75.6
77.6
78.5
76.0
79.5
72.9
72.8
68.5
77.8
Unicoder [28]
85.6
81.1
82.3
80.9
79.5
81.4
79.7
76.8
78.2
77.9
77.1
80.5
73.4
73.8
69.6
78.5
XLM-Raa
85.4
81.4
82.2
80,3
80.4
81.3
79.7
78.6
77.3
79.7
77.9
80,2
76.1
73.1
73.0
79.1
XLM- Rage
89.1
85.1
86.6
85.7
85.3
85.9
83.5
83.2
83.1
83.7
81.5
83.7
81.6
78.0
78.1
83.6
Cruss Trunsfer:
tune multilnguul model English training set
XLM [2]
85.0
78.7
78.9
77.8
76.6
77.4
75.3
72.5
73.1
76.1
73.2
76.5
69.6
68.4
67.3
75.1
Unicoder [X)
85.1
79.0
79.4
77.8
77.2
77.2
76.3
72.8
73.5
76.4
73.6
76.2
69.4
69.7
66.7
75.4
M-BERT IMI
82.1
73.8
74.3
71.1
66.4
68.9
69.0
61.6
64.9
69.5
55.8
69.3
60.0
50.4
58.0
66.3
UD-BERT
82.7
74.9
75.2
72.0
67.4
69.2
70.3
62.7
65.8
70.3
59.6
69.7
61.4
51.2
S8.7
67.4
85.8"
779.7
RITT
78.7
777.5
79.6
"8..1
74.
75.8"
"Y6.3"
74.6
7677"
772.3
86.55
68.3"
766.2
UD-XLM-
86.5
80.3
81.6
79.8
78.4
80.0
78.9
75.1
74.4
77.1
75.2
77.3
73.0
67.0
68.8
76.9
XLM-R
89.1
84.1
85.1
83,9
82.9
81.2
79.6
79.8
80.8
78.1 1
80.2
76.9
73.9
73.8
80.9
UD-XLM-
89.4
84.8
85.6
84.5
83.6
84.7
81.2
80.0
81.0
81.9
78.6
80.7
76.8
74.4
74.3
81.4
Second, our UD-BERT and UD-XLM-R performed better in most cases of the 14 transferring target languages. UD-BERT has an average increase of 1.1 when compared to the baseline, UD-XLM- Rhose has an average increase of 0.7, and UD-XLM-R has an average increase of 0.5. This improvement highlights the fact that the cross-lingual transferring ability of our multilingual PrLM has improved as a result of the employment of universal linguistic structures. Furthermore, the cross-lingual transfer effect of our UD-XLM-R outperforms BERT with a similar model struc- ture and parameters, who translated the test set for evaluation. The results of our UD-XLM-R,, R achieved better results than all the methods which leveraging a monolingual language model on the translations, which shows that cross-lingual transfer is more promising mode.
Text classification is relatively simple and intuitive NLU task. To further verify our conclusions, we conducted experiments on a more complex task- cross-lingus Machine Reading Comprehen- sion (MRC). The results on XQuAD dataset are shown in Table E Similarly, we first compare the MRC results on the source language English. The performance of UD-BERT and UD-XLM-R has increased relative to the baseline, which verifies the conclusion that our universal linguistic structure improves the NLU ability of multilingual language model. For the 11 transferring target languages, we observed a similar improvement trend on UD-BERT and UD-XLM-R as in the XNLI task, and the improvement was even greater. The average improvement of UD-BERT, UD-XI DD-MLM-R base and UD-XLM-R large reached 1.0, 0.6, 0.8 Fl scores respectively. Among them, Arabic has the largest improvement, with 1.3, 2.3, and 6.6 F2 scores respectively. All the results here reveal that universal syntactic structure information embedded is effective for cross-lingual MRC task.
Universal Linguistic Structure Parsing Since the universal dependency parsing structure and the dependency parse encoding structure are built into our multilingual PrI.M, to demonstrate that our model learned how to extract linguistic structure and to encode the structure into the output
7

---

Table 2: F2 EM scores XQuAD with English as the source language for each target language.
Cross- Transfer
em
ar
es
ru
th
tr
zh
ro
Avg
Fj
83.5
61.5
70.6
62.6
75.5
59.2
71.3
42.7
55.4
69.5
58.0
72.7
65.2
m-BERT
EM
72.2
45.1
54.0
44.9
56,9
46.0
53.3
33.5
40.1
49.6
48.3
59.9
50.3
Fi
83.9
62.8
72.3
62.9
75.7
59.0
71.6
48.6
55,5
69.9
58.7
73.7
66.2
UD-BERT
EM
72.5
46.3
56.0
45.8
57.3
46.3
54.0
38.5
40.2
49.7
48.6
60.3
51.3
83.6
66.8
74.4
73.0
76.4
68.2
74.3
66.5
68.3
73.7
51.3
77.8
71.2
XLM-Rrus
EM
72.1
49.1
55.7
58.3
51.7
58.1
56.7
52.8
53.8
42.0
62.8
F1
84.0
69.1
74.9
73.5
77.0
68.4
74.3
66.9
69.3
74.1
51.8
78.0
UD-XLM-R
EM
72.5
51.2
60.5
56.0
58.3
57.1
52.9
42.6
63.3
F1 86.5
68.6
80.4
79.8
82.0
76.7
80.1
74.2
75.9
79.1
59.3
83.6
77.2
XI.M-Ringe
EM
75.7
49.0
63.4
61.7
63.9
59.7
64.3
62.8
59.3
59.0
50.0
69.7
61.5
86.8
75.2
80.9
80.0
82.3
77.1
80.3
73.8
76.3
79.5
59.4
83.9
78.0
UD-XLM-RI
EM
75.9
58.2
63.8
61.7
64.0
59.8
64.5
62.2
60.5
59.8
49.9
69.9
Table 3: The monolingual UD parsing results (UAS/LAS) on 22 UD Treebanks. means that the
PrLM is tested directly without additional training of a biaffine dependency parsing model.
AI-FFT
m
BERT
UD-BERT
UD-BERT
XLM- Ruse LAS
UD-XLM-
Rose
XLM-
KLu-RRuge
UD-XLM MLM-Rurge
UAS
LAS
UAS
LAS
UAS
LAS
UAS
UAS
LAS
UAS
LAS
UAS
bg
94.75
90.88
95.76
91.57
96.12
93.09
96.42
93.56
96.57
93.80
96.53
93.72
96.69
94,04
ca
95.36
93.25
94.97
92.72
95.61
94.27
95.53
94.26
95.78
94.61
95.75
94.58
95,94
94.72
94.38
91.62
95.11
91.94
95.62
93.17
95.60
93.30
95.82
93.43
95.87
93.69
95.99
93.87
nl
94.74
92.72
94.44
91.27
95.38
93.61
95.36
93.44
95.79
93.82
95.63
93.78
96.13
94.50
92.52
91.29
91.34
88.17
93.01
91.43
93.60
91.83
94.15
92.55
93.47
91.73
94.19
92.54
et
90.88
88.95
90.04
86.23
91.65
89.73
92.53
90.78
92.78
91.02
93.16
91.50
93.28
91.68
fi
92.98
90.65
89.41
82.51
94.07
91.89
94.99
93.14
95.15
93.44
95.17
93.46
95.66
94.01
fr
94.12
90.75
95.52
92.20
96.08
94.24
96.10
94.34
96.48
94.69
96.01
94.15
96.53
94.75
90.77
86.83
88.64
81.78
91.08
87.35
91.30
87.42
91.58
87.63
91.39
87.50
91.72
87.94
92.32
89.95
92.52
88.55
93.45
91.06
93.50
91.41
93.87
91.64
93.67
91.48
93.99
91.79
hi 96.54
94.22
95.25
91.34
96.71
94.41
96.73
94,51
97.03
94.99
96.93
94.76
97.17
95.09
id
88.29
83.97
87.73
79.18
87.96
83.72
88.25
84.06
88.51
84.25
88.38
84.27
88.48
84.19
it
95.63
94.01
95.99
93.59
96.32
95.16
96.15
94.93
96.70
95.33
96.26
95.01
96.82
95.60
ko 90.73
88.27
81.90
71.29
90.99
88.67
91.33
89.25
91.42
89,30
92.15
89.79
92.16
89.98
la 85.69
81.84
84.57
79.04
87.01
83.45
86.64
82.99
89.86
86.91
86.97
83.29
90.44
87.33
Iv 91.55
89.06
90.57
85.65
92.06
89.72
93.53
91.25
93.88
91.50
94.23
91,90
94.56
92.52
95.62
93.88
94.61
92.64
96.23
95.17
96.57
95.58
96.71
95.63
96.70
95.64
96.74
95.75
pl 98.15
96.54
96.26
80
98.54
97.14
98.51
97.22
98.80
97.77
98.47
97.08
98.79
97.74
92.70
86.39
93.58
87.38
94.12
89.39
94.25
89.67
94.60
90.11
94.39
89.85
94.73
90.32
ru 95.26
94.00
95.30
92.95
95.74
94.58
96.25
95.26
96.54
95,59
96.38
95.45
96.62
95.72
sk
94.93
91.40
96.22
93.19
96.62
93.95
95.96
93.21
97.49
95.53
95.56
92.74
97.26
95.34
es
94.69
92.89
94.08
90.72
94.92
93.23
94.98
93.44
95.38
93.80
95.33
93.80
95.55
94.09
Avg
93.30
90.61
92.45
87.82
94.06
91.75
94.28
92.04
94.77
92.61
94.47
92.24
94.97
92.89
representations, we performed the universal linguistic structure parsing on UD and multilingual constituent parsing tasks. It is worth noting that, to shield the influence of the increase in the number of parameters caused by the addition of the PrLM to the downstream task model, kept all the PrLM parameters frozen in the universal NLU evaluation.
In Table 3, we evaluated the effect of using the multilingual PrLM to enhance the parsing model on 22 languages of UD dataset respectively, in order analyze how many features useful (i.e., syntactic- aware features) for parsing are provided by the output representation of the various multilingual PrLM. At the same time, to verify that our model has learned syntax, we listed the results that UD-BERT is tested directly 22 languages without additional training of biaffine dependency model.
Comparing UD- UD-BERT and m- m-BERT, our UD-BERT obtained generally better results, except for Indonesian, with a 0.76/1.14 14 UAS/LAS average improvement. The average score of UD-BERT" is comparable to that of m BERT, and it outperforms m-BERT in some languages, indicating that our model has integrated well-performing parser that can be directly applied to practice without additional training. When we further compare UD-BERT to stronger XLM-R and XLM-R baselines, we found that XLM-R achieved similar results as UD-BERT. Since XLM-R uses more data and longer pre-training time, from this point of view, UD-BERT has played role in reducing
8

---

Table 4: Labeled F scores on PTB, CTB, and SPMRL test sets.
AlI-FT
en
ar
eu
fr
de
he
hu
ko
pl
zh
Avg
M-BERT
94.87
88.60
91.06
84,98
90.20
83.35
92.33
89.36
96.36
83.05
90.83
89.54
UD-BERT
95.09
89.97
92.23
85.57
91.06
84.36
93.46
89.72
96.59
85,06
90.99
90.37
XLM-Rauee
95.50
89.67
91.68
85.63
91.54
85.20
93.92
90.82
96.72
86.00
91.84
90.77
UD-XLM-R
95.83
90.22
92.67
86,10
91.98
86.03
94.61
91.32
96.95
86.71
92.30
91.34
XLM-Range
95.74
90.41
92.54
86.47
92.10
86.11
94.56
91.77
97.15
86.80
92.24
91.44
UD-XLM-Rau Rurgge
96.15
90.77
94.13
86.96
92.65
86.97
95.41
92.16
97.33
89.06
92.43
92.18
Table 5: Unlabeled F scores on PTB, CTB, and SPMRL test sets.
Cross-Transfer
em
ar
eu
fr
de
he
hu
ko
pl
SV
zh
Avg
m- m-BERT
95.54
24.22
32.61
59.40
44.36
46.01
57.86
33.99
37.47
71.69
56.87
50,91
UD-BERT
95.93
27.09
35.50
60.87
45.24
46,88
58,99
35.67
39.23
72.20
57.74
52.30
XLM-Rusc
96.19
28.17
34.34
59.70
44.98
47.05
60.26
38.95
38.06
73.68
55.36
52.43
UD-XLM-Ru
96.60
30.46
37.75
61.36
45.52
47.42
62.00
42.94
39.83
73.90
59.97
54.34
XLM-Ruse 96.44
20.36
34.23
59.8I
45.04
47.01
59.37
36.72
37.87
73.43
34,34
31.35
UD-XLM-Rua.
96.73
23.44
37.63
61.18
45.71
47.30
60.46
41.26
38.42
73.81
59.58
53.23
the data and time required for pre-training due to the addition of syntactic supervision information. Further comparing UD-XLM-R with the baseline XLM-R, we found that our method is still helpful on such strong baselines, which improves the parsing performance in each language. In addition, we also evaluate the improvement of our model zero-shot pross-lingual' setting and in low-resource languages. The results are shown in Appendices A.3 and A.4 respectively.
Though verified the linguistic feature extraction and encoding capabilities of our PrLM in the UD parsing, since the evaluation task is the same as in the pre-training, it is hard to illustrate this effect on the cross-task linguistic parsing task. Thus, conduct further exploration constituent parsing benchmarks. Table Hl shows the enhanced ability of the language model for mono- lingual constituent parsing. The comparison demonstrates that with the help of additional universal dependency features in the representations, the constituent parsing performance of UD-BERT and UD-XLM-R has been greatly increased, and the average improvement has reached 0.83, 0.57, and 0.74 respectively. This reflects that the universal syntactic features contained in the representations are very helpful for downstream tasks. In addition, with our syntax- aware multilingual PrLM, the integration of syntactic tree is no longer necessary to change the downstream task model for adding extra syntactic encoders, which will greatly reduce the cost of syntactic tree information application.
multilingual
The multilingual boosting ability of our PrLM is shown in Table AL Similarly, we evaluated the PrLM's cross-lingual transfer ability on constituent parsing in Table 5L and because there are no available universal constituent label annotations, we only report unlabeled F1. Again, our UD- BERT and UD-XLM-R achieve an improvement in all transferring target languages, which shows that the explicit universal features contained in our PrLM can be used to help downstream cross- lingual transfer tasks. Since the universal dependency structure can be derived directly by additional inference in our PrLM's intermediate output, our good tross-lingua transfer ability is interpretable. Furthermore, our studies found that the transfer effect of XLM-Rpase superior to that of XLM- Runge implying that larger model parameters may not always result in higher cross-lingual transfer capabilities.
5 Ablation Study
In our multilingual PrLM, we propose both novel model structure and novel pre-training strategy, whose effectiveness is verified by the ablation study in Table G. We also doubt that the gain is partly derived from the additional UD plain text, although the amount of UD plain text is small. So we also show the results of training the baseline with additional UD plain text (w/ UD plain text) in Table R From the results, if universal dependency parsing is only used as an additional pre-training process (wo US Integration), there is only slight performance improvement compared to the baseline,
In the strict sense, this cannot be totally categorized as zero-shot because the language model part of the full parser has been pre-trained by UD annotations.

---

which shows that just as -re-iiaining approach is not enough to fuse the universal syntax implicitly. When we only employ the model structure without UD pre-training (w/o UD Pre-training), this additional parser structure, even though it is not trained by the UD supervision, can still improve the performance, and the improvement greater than that of w/o US Integration. Comparing m-BERT and m- (-BERT(w/ UD plain tex(), we find that the performance difference between them is very small, and the performance is even lower after the addition of UD plain text. This indicates that a small amount of UD plain text can not directly improve the model. Combining the above results with that our full UD-BERT, we can conclude that the supervised learning the UD structure and its explicit integration into the representation
indispensable options for performance improvement. And our universal structure integration can help the model learn the syntactic knowledge without additional supervision and improve the performance because it has accountable form that can well model the syntactic structure features.
Table 6: Ablation studies on XQuAD and SPMRL.
XQuAD(Cross-Trassfff
SPMRL(AI-FT)
Method
ar
de
en
el
ar
de
en
eu
Fi
EM
Fi
EM
Fi
EM
F1
EM
Fi
Fi
Fi
Fj
m-BERT
61.5
45.1
54.0
83,5
72.2
62.6
44.9
88.6
90.2
94.9
UD plain
61.0
44.6
70.1
53.8
82.9
71.8
62.3
44.8
88.5
90.2
94.6
91.0
UD-BERT
62.8
46.3
72.3
56.0
83,9
72.5
62.9
45.8
90.0
91.1
95.1
92.2
w/o US Integration
61.9
45.4
70.9
54.5
83.8
72.6
62.5
45.0
88.9
90.4
94.8
91.3
w/o UD Pre-training
61.5
45.0
70.7
54.3
83.6
72.2
62.7
45.2
89.1
90.6
94.9
91.5
6 Conclusion
In this work, we propose multilingual PrLM that supports both explicit universal structure learning and implicit language modeling. We chose the universal dependency parses as our universal structure knowledge and evaluate the cross-lingual modeling capability of our model on two rross-lingual NLU tasks and four syntactic parsing datasets. Our model outperforms m- m-BERT and XLM-R in all tasks and achieves state-of-the-art results syntactic parsing. Unlike other works that use the syntax as an objective or feature, our structure learning is used not only for a pre-training objective but also for improving the representation, which makes model both PrLM and universal dependency parser. This will greatly change the way that downstream NLP tasks use syntax, because have explicitly integrated the syntactic knowledge into the representation of the Pr..M.
References
[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers). Minneapolis, Minnesota: Association for Com- putational Linguistics, Jun. 2019, pp. 4171-4186.
[2] Y. Sun, S. Wang, Y. Li, S. Feng, Tian, H. Wu, and H. Wang, "Ernie 2.0: continual pre- training framework for language understanding." in Proceedings of the AAAI Conference Artificial Intelligence, vol. 34, no. 05, 2020, pp. 8968-8977 8975.
[3] I. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang. J. Gao, M. Zhou, and H.-W. Hon, "Unified language model pre-training for natural language understanding and generation." in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: 120b2za925500aaa17771111555555 Patp:// Paper.pdf
[4] Y. Liu, M. Ont, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and Stoyanov, "Roberta: robustly optimized BERT pretraining approach," CoRR, vol. abs/1907. 1692, 2019. [Online]. Available: http://arxn.oorraass////...11
10

---

[5] D. Cer, M. Diab, E. Agine, I. Semantic textual similarity multilingual and crosslingual focused evaluation," in Proceedings of the 11th International Workshop Semantic Evaluation (SemFvval-007). Vancouver, Canada: Association for Computational Linguistics, Aug. 2017, pp. 1-14. [Online]. Available:
and L. Specia, "SemEval-2011 task 1:
[6] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ questions for machine comprehension of text," in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguisties, Nov. 2016, pp. 2383-2392. [Online]. Available: http://www.cllwbbbrgaaatttng///
[7] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng. and C. Potts, "Recursive deep models for semantic compositionality over a sentiment treebank," " in Proceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1631-1642.
[8] J. Zhou, Z. Li, and H. Zhao, "Parsing all: Syntax and semantics, dependencies and spans," in Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Associa- tion for Computational Linguistics, Nov. 2020, pp. 4438 4449.
[9] A. Conneau and G. Lample, "Cross-lingual language model pretraining," in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Aché-Buuc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 7057-7067. [Online]. Available: httpp::/proccedingg.eerrrr Ahsssrchh
[10] A. Conneau, S. Wu, H. Li, L. Zettlemoyer, and V. Stoyanov, "Emerging cross-lingual structure in pretrained language models," Proceedings of the 58th Annual Meeting of the Asvociation for Computational Linguistics. Online: Association for Computational I. Linguistics, Jul. 2020, Available:
pp. 6022-6034.
[11] W. Wang. Bi, M. Yan, C. Wu, J. Xia, Z. Bao. L. Peng, and L. Si. "Structbert: Incorporating language structures into pre-training for deep language understanding." in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. [Online]. Available:
http:///ppaneeve..eeffonmmm' (ggg//o
[12] J. Zhou, Z. Zhang. H. Zhao, and S. Zhang, "Limit-bert: Linguistics informed multi-task bert," in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
[13] R. Caruana, "Multitask learning: knowledge- based source of inductive bias," Proceedings of the Tenth International Conference on Machine Learning. Morgan Kaufmann, 1993, pp-
41-48.
[14] S. R. Bowman, J. Gauthier, A. Rastogi, R. Gupta, C. D. Manning, and C. Potts, "A fast unified model for parsing and sentence understanding." " in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1466-1477. [Online], Available:
[15] G. Angeli, M. J. J. Premkumar, and C. D. Manning, "Leveraging linguistic structure for open domain information extraction," " in Proceedings the 53rd Annual Meeting the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), 2015, pp. 344-354.
[16] K. Toutanova, X. Lin, W.-t. Yih, H. Poon, and C. Quirk, "Compositional learning of em- beddings for relation paths in knowledge base and text," in Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), 2016, pp. 1434-1444.
[17] R. McDonald, J. Nivre, Y. Quirmbach-Brundagge Y. Goldberg, D. Das, K. Ganchev, K. Hall, S. Petrov, H. Zhang, O. Täckström, C. Bedini, N. Bertomeu Castelló, and J. Lee, "Universal Dependency annotation for multilingual parsing," in Proceedings of the 51 Annual Meeting of the Association for Computational I Linguistics (Volume 2: Short Papers). Sofia, Bulgaria: Association for Computational Linguistics, Aug. 2013, pp. 92-97.
[18] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, Guzmán, Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, "Unsupervised cross-lingual representation learning
11

---

at scale," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440-8451.
[Online]. Available:
[19] D. Seddah, S. Kübler, and R. Tsarfaty, "Introdueing the SPMRL 2014 shared task on parsing morphologically-rich languages." in Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non- Con-Canonical Languages. Dublin, Ireland: Dublin City University, Aug. 2014, pp. 103-109. [Online].
Available:
14-6111
[20] M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz, "Building a large annotated corpus of English: The Penn Treebank," Computational Linguistics. vol. 19, no. 2, pp. 313-330, 1993. [Online]. Available:
[21] N. Xue. F.-D. Chiou, and M. Palmer, "Building large-scale annotated Chinese corpus." in COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
[22] W. Wang. B. Bi, M. Yan, C. Wu, J. Xia, Z. Bao, L. Peng, and L. Si. "Structbert: Incorporating language structures into pre-training for deep language understanding." in International Conference on Learning Representations, 2020. [Online]. Available: https:/openneviewnne//mmm (igg:/ooummndddBBB(IISSS
[23] X. Ren, X. Sun, H. Wang, and Q. Liu, "Towards semantics-enhanced pre-training: Can lexicon definitions help learning sentence meanings?" 2021.
[24] E. Strubell, P. Verga, D. Andor, D. Weiss, and A. McCallum, "Linguisticall--inforeed self-attention for semantic role labeling," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 5027-5038. [Online]. Available: https///ccannhhlooy..r///// Drg//D8--/DD1
[25] D. Sachan, Y. Zhang, P. Qi, and W. L. Hamilton, "Do syntax trees help pre-trained transformers extract information?"
in Proceedings of the 16th Conference of the European Chupter of the Association for Computational Linguistics: Main Volume. Online: Association for Computational Linguistics, Apr. 2021, pp. 2647-2661. [Online]. Available: https://aclantoolgy.ooo 2021.e2cl-maaamiee
[26] H. Huang, Y. Liang, N. Duan, M. Gong, L. Shou, D. Jiang, and M. Zhou, "Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks," in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)), Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 2485-2489999 2494.. [Online]. Available: http://www...llweoorrrrtttooo///
[27] W. Ahmad, H. Li, K.-W. Chang, and Y. Mehdad, "Syntax-augmente multilingual BERT for cross-lingual transfer," in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). Online: Association for Computational Linguisties, Aug. 2021, pp. 4538-4554. [Online]. Available:
[28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https: 1448335331444444 b Paper.pdf
/proceedings.neumm http://mnssnnnuussssscpppppppeee
[29] T. Dozat and C. D. Manning, "Deep biaffine attention for neural dependency parsing," ICLR, vol. abs/161 1. 1.01734. 2017.
[30] J. Nivre, M.-C. de Marneffe, F. Ginter, J. Hajič, C. D. Manning, S. Pyysalo, S. Schuster, F. Tyers, and D. Zeman, "Universal Dependencies v2: An evergrowing multilingual treebank collection." in Proceedings of the 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 4043-4043. [Online]. Available: http:://www..eewebooogoanhhho//p
[31] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa,
12

---

A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, "Google's ' neural machine translation system: Bridging the gap between human and machine translation," CoRR, vol. abs/1609.08 (06.999918000 010000884444 144, 2016. [Online]. Available: http://axiiv.org/bb///oo 1609.08144
[32] T. Kudo and J. Richardson, "Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, E. Blanco and W. Lu, Eds. Association for Computational Linguistics, 2018, pp. 66-71. [Online]. Available:
htpss//to.orr/11118633/////1
[33] D. Kingma and Ba, "Adam: A method for stochastic optimization," in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Bengio and Y. LeCun, Eds., 2015. [Online]. Available:
http://rxiv.org/abs// (11.////b///1..988..
[34] D. Hendrycks and K. Gimpel, "Bridging nonlinearities and stochastic regularizers with gaussian error linear units," CoRR, vol. abs/1606.08415. 2016. [Online]. Available:
http://axxv.orr/ab///bb...8
[35] A. Williams, N. Nangia, and Bowman, "A broad coverage challenge corpus for sentence understanding through inference," in Proceedings of the 2018 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume I (Long Pupers). New Orleans, Louisiana: Association for Computational Linguistics. Jun. 2018, pp. 1112-1122. [Online]. Available:
[36] M. Artetxe, S. Ruder, and D. Yogatama. "On the cross-lingual transferability of monolingual representations," Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 4623 4637. [Online]. Available: http://www.aawwbnaaaatayy///aaaa
[37] Y. Zhang, H. Zhou, and Z. Li, "Fast and accurate neural erf constituency parsing," in Proceedings of the TwentyNinth International Joint Conference on Artificial Intelligence, IJCAI-20, C. Bessiere, Ed. International Joint Conferences on Artificial Intelligence Organization, 7 2020, pp. 4046-4053, main track. [Online]. Available: https://dos.org//u) 20025/1)
13


Reviews:
[{"review_id": "Z2vpFCJ7KVX", "reviewer": "Reviewer_Cgr3", "summary": "This paper proposes a multilingual pretrained language model that learns from universal dependency parsing as well as language modeling. The UD layer is done on top of the transformer by following a biaffine dependency parsing (Dozat and Manning.) and the language modeling is learned with a MLM layer as in BERT/RoBERTa.  For MLM they use 104 languages while for UD learning they use 60 languages present in the Universal Dependencies project. The evaluation is done in XNLI, XCuad, UD parsing, SPMRL constituency parsing (SPMRL was a workshop on parsing for morphologically rich languages, in this case they use the constituent version of those), PTB and CTB\n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is similar to Swayamdipta et al. 2019 https://arxiv.org/pdf/1908.11047.pdf. In contrast to the findings in Swayamdipta et al. 2019, their model outperforms multilingual-BERT and XLM-R, on cross-lingual natural language understanding benchmarks and parsing tasks.\n\nThe evaluation is done in XNLI, XCuad and UD parsing. The UD parsing evaluation is definitely boosted given that the pretraining has seen that data.\nLine 196: why not separating SPMRL as another dataset?\n\nLine 25:  \"some work has shown that, m-BERT does not have equally high-quality representation for all of the languages it includes and for some languages, m-BERT performs worse than non-BERT\" --> does your work address this issue?\n\n- “multilingual PrLMs only focus on plain text and ignore obvious 6 universal linguistic structure clues” --> There is a body of work showing that multilingual pretrained LMs are able to learn syntax without supervision.  (Hewitt and Manning. NAACL 2019) (Perez-Mayos et al. EACL 2021) (Maudslay and Cotterell 2021) among others. You should cite this work and contrast with your approach.\n\nLines 35-36: those are 2015 and 2016 papers that show the effectiveness of parsing in downstream applications. Is there something a bit more recent?\n\nLines 168-169: how were the number of layers selected?\n\nFigure 1 is not very clear. Also why is it not part of the main paper? Have you tried just sharing the weights between the two tasks instead of having the US integration. This part is not properly motivated from the write up.\n\nMinor comment: The math in the paper is not always properly written (e.g. line 143, that equation seems lost) I find this guide very useful: https://web.cs.ucdavis.edu/~amenta/w10/writingman.pdf\n\nTables 1, 2, 3 and 4 : some of the results in bold are not the best results. E.g. First column RoBERTa is 91.3 while your model is 89.4. In table 2 some columns do not have result in bold when those are from the model without syntax. For some reason table 4 does not have bold scores.\n\nTables 1, 2, 3 and 4: I would recommend reporting an average score across 5 models trained on different random seeds and the standard deviation. Many of the results here are very similar to each other, and it is not clear whether there is any statistical significance.\n\nI am glad to see Section 5, but it  would significantly improve by probing the linguistic knowledge of the generated pretrained LM following (Hewitt and Manning .2019 (https://aclanthology.org/N19-1419/), not only the score in downstream applications.\nTable 7: why ar and eu? why not other languages?\n\nworth citing: https://arxiv.org/abs/1904.02099\n\nThe paper does not explain how they manage to interleave the training between the two main tasks. In one of them they have millions of sentences, in the other (UD) way less. How do you ensure that there is no catastrophic forgetting  or overfitting on the smaller dataset during training?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Minor call-out: The authors are not reporting the total amount of compute and the type of resources used. It is likely that this kind of work cannot be replicated by most academic labs.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1h 10 mins"}, "extra_text": {}}, {"review_id": "YVWYH9XxlVx", "reviewer": "Reviewer_WQqW", "summary": "This paper proposes a multilingual language model pre-training strategy which incorporates both universal dependency parsing and masked language modeling objectives. The authors integrate two additional layers, namely Universal Structure Learning layer and Universal Structure Integration, to mBERT and XLM-R for explicitly learning the language structures representations. The experimental results show that the proposed model improves both mBERT and XLM-R in multiple cross-lingual tasks, including Cross-lingual Natural Language Inference (XNLI), Cross-lingual Question Answering (XQuAD), and Universal Linguistic Structure Parsing.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n- Incorporating language structures into language model pre-training has been explored in the monolingual setting[1,2]. This work extends this idea to the multilingual language model pre-training, and shows the improvement in cross-lingual tasks. However, it would be helpful to highlight the differences other than the pre-training data.\n\nQuality: \n- The proposed method is technically sound. The claims are supported by extensive experiments in multiple well-known datasets. To further strengthen the paper, the authors could evaluate their models on XTREME [3].\n\nClarity: \n- This paper is well organized. However, some training details such as learning rate and training batch size are missing. It is also not cleat how the authors balance the training for USL and MLM as they are trained with different data.\n\nSignificance: \n- Learning robust multilingual representation is an important task. This paper achieved strong empirical results in many cross lingual tasks.\n\nAdditional Questions (feedback):\n- Why both mBERT and XLM-R are trained from scratch? What is the performance if both models are initialized with original weights?\n- The ablation study is only conducted on a linguistic structure parsing dataset (SPMRL), and the authors reach the conclusion that \"the additional parser structure, even though it is not trained by the UD supervision, can still improve the performance, and the improvement is greater than that of w/o Universal Structure Integration\". However, I would like to see the ablation experiments on cross-lingual NLU benchmarks such as XNLI to validate this conclusion.\n\n**Post-rebuttal updates** The authors response has addressed some of my concerns (rating 5 --> 6). However, I expect authors can conduct additional ablation study on XNLI and XQuAD to verify the effectiveness of Universal Structure Integration and UD Pretraining.\n\n[1] StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding. ICLR 2020\n[2] LIMIT-BERT : Linguistics Informed Multi-Task BERT. EMNLP 2020\n[3] XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. ICML 2020\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}, {"review_id": "PXbTpfY9XT", "reviewer": "Reviewer_dbNQ", "summary": "This paper proposes an improvement of multi-lingual pre-trained language models (PrLMs) by incorporating the dependency knowledge from the universal dependency (UD) treebank (v2.2).  A universal structure (US) learning layer and a US integration layer are added on the top of tranformer-based PrLMs.  In the US learning layer, two MLPs are used to convert the contextualized embedding from the PrLMs to representations of a word as heads or dependencies.  Then another MLP is used to predict the (arc, label) for each of the (head, dep) pairs.  In the US integration layer, the representation learned from the US learning learning are integrated with the origial repsresentations output from the PrLMs.  Such denendency enhanced PrLMs are trained with mBERT-base, XLM-Roberta-base and XLM-Roberta-large.  In additional to the original training data for these models, this paper add the data from UD treebank v2.2 with 60 languages, both for MLM training and US prediction.  Experiments are done in various tasks: XNLI, XQuAD and Universal Linguistic Structure Parsing with test data from UD treebank v2.7, SPMRL Shared Task 2014, as well as PTD (English) and CTB (Chinese).  Results show the proposed models outperforms the baseline models in almost all of the tested tasks.  Ablation study results are given on language AR and EU for SPMRL datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper is in general well-motivated and clear.  \n\nMy main concern is whether or to what extent the improvements are really brought by the dependency knowledge.\nThe ablation study is not very convincing to me, because it is done only on a syntactic parsing task in two languages.\nIt is reasonable that the introducing of UD data can improve the performance of parsing tasks.\nCould it also improve the performance of non-parsing tasks?  You should do ablation study in other non-parsing tasks, such as XNLI and XQuAD.\n\nFurther, the improvement is possibly brought by the additional parameters and training data of the model, rather than the dependency information.\nCould you give the number of models of the proposed models and all baseline models?\n\nOther suggestions:\n\nFigure 1 (model structure) in the appendix should be included in the main paper for better understanding.\nThe second equation under line 106: k appears in the left hand side but not in the right hand side.\n\n---------------------\n\nAfter rebuttal:\n\nThe new ablation study results partially answer my concerns, so I will raise my rating to 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "CJ7vCfQiBPR", "reviewer": "Reviewer_RgYB", "summary": "This paper presents a method to pretrain large-scale masked language models while integrating syntactic information from parses. Their experiments and analysis show that their method is effective in a range of crosslingual transfer tasks.  The pretraining objective involves predicting dependency parsing in addition to masked language modeling, which is commonly used in many pretrained models. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- This paper presents a straightforward way of integrating parsing information into a pretraining objective. They proposed a staged training strategy that is well motivated to address parsing failure in an early stage. \n\n**Weaknesses**\n\n**Updates after the author response**\n1. I sincerely apologize for missing the XML-R large rows in the tables. All questions except the last one are fully addressed. \n2. Regarding the last question, I agree that making good use of existing UD corpora is a promising and important direction. And as the authors point out in the response, it's very encouraging to see that some languages without parsing data still benefit from their model. This makes me think that it'd be informative to analyze results by breaking down the improvements by whether or not each language is included in the UD datasets (or more thoroughly, the size of the UD corpus for each language to see the effect of the amount of parsing data). \n3. As other reviewers suggest, there is still a concern about the assessment of the effect of parsing pretraining. Ablation studies would be strengthened if they were done on downstream tasks. Moreover, since UD corpora can provide additional pretraining data even without UD annotations, it can be useful to add the text data to the pretraining data for the XLM baselines as well. For example, it's possible that the text data from the specific domain, not just the UD parses, is contributing to the improvement over the XLM baselines (though this is perhaps only critical in UD parsing results). \n4. I updated my score accordingly. \n----\n\n- I have several critical questions for the authors to be addressed. In particular, I have questions regarding their experimental settings and empirical comparisons. Please see them below. \n\n**Questions for the authors**\n- In Table 1, why is there no XLM-R large in the crosslingual setting? The authors make comparisons for English between XLM-R large and the corresponding UD-XLM-R large, but do not compare XLM large in the crosslingual setting. The All-FT results suggest that XLM-R large substantially would outperform XLM in crosslingual transfer as well, but no head-to-head comparison is made between UD-XLM-R large and XLM-R large for crosslingual transfer. \n- And numbers in bold face for UD-XLM-R large are confusing; since the corresponding XLM-R large is not evaluated in the crosslingual setting, their results are not truly comparable to other models with limited capacity. \n- Conversely in Table 1, have you tried your models like UD-XLM-R large in the All-FT setting? Did it perform equally well compared to XML-R large? I wonder if the proposed method can work in high resource settings as well. This would help us understand whether the proposed model is only effective in low-resource settings.\n- Table 2 also has the same problem: while numbers from XLM-R large are in bold face, there is no head-to-head comparison. \n- Interestingly, XLM-R and UD-XLM-R are compared in Tables 4-6 for other tasks. This makes one wonder more why it is not done in the other tables. \n- I might be missing this, but what (labeled) data is used for $L_{USL}$ training (line 4 in Algorithm 1)? Did you use the UD corpora? Or did you use automatically parsed data from the UD treebank? The difference in training data is important to understand what is needed for the proposed method and whether comparisons of different models are fair. For example, If UD parsing data is necessary, whether or not it is automatically parsed or parsed by human annotators, the proposed method is not expected to be practical in many truly low-resource languages whose parses/parsers are not readily available. Since the paper emphasizes results from crosslingual transfer, this is an important point. \n \n**Typos and Minor Issues**\n- Line 44: \"meantime, we propose and structural encoder to encode\" -> \"meantime, we propose a structural encoder to encode\".\n- Line 55: \"mono-lingual\". Please be consistent in hyphenation throughout the paper. \n- Lines 265-266: run-on sentence. \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper breaks new ground by adding explicit structural supervision to a multilingual pretrained LM, and looking at the interaction between structural supervision (which is known to be helpful) and multilinguality. Overall the work is carefully done and well motivated, and the results are strong, indicating that “universal” structural information improves multilingual transferability. This is an important and necessary, but not overly surprising, next step. The risk of “obvious” papers like this is that they are done poorly, but this paper does a good job (although the AC would encourage the authors to include the follow-up results on XNLI/XQuAD tasks whenever the paper ends up being published).
Decision: Accept (Poster)