Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an improvement of multi-lingual pre-trained language models (PrLMs) by incorporating the dependency knowledge from the universal dependency (UD) treebank (v2.2).  A universal structure (US) learning layer and a US integration layer are added on the top of tranformer-based PrLMs.  In the US learning layer, two MLPs are used to convert the contextualized embedding from the PrLMs to representations of a word as heads or dependencies.  Then another MLP is used to predict the (arc, label) for each of the (head, dep) pairs.  In the US integration layer, the representation learned from the US learning learning are integrated with the origial repsresentations output from the PrLMs.  Such denendency enhanced PrLMs are trained with mBERT-base, XLM-Roberta-base and XLM-Roberta-large.  In additional to the original training data for these models, this paper add the data from UD treebank v2.2 with 60 languages, both for MLM training and US prediction.  Experiments are done in various tasks: XNLI, XQuAD and Universal Linguistic Structure Parsing with test data from UD treebank v2.7, SPMRL Shared Task 2014, as well as PTD (English) and CTB (Chinese).  Results show the proposed models outperforms the baseline models in almost all of the tested tasks.  Ablation study results are given on language AR and EU for SPMRL datasets.
main_review: 
The paper is in general well-motivated and clear.  

My main concern is whether or to what extent the improvements are really brought by the dependency knowledge.
The ablation study is not very convincing to me, because it is done only on a syntactic parsing task in two languages.
It is reasonable that the introducing of UD data can improve the performance of parsing tasks.
Could it also improve the performance of non-parsing tasks?  You should do ablation study in other non-parsing tasks, such as XNLI and XQuAD.

Further, the improvement is possibly brought by the additional parameters and training data of the model, rather than the dependency information.
Could you give the number of models of the proposed models and all baseline models?

Other suggestions:

Figure 1 (model structure) in the appendix should be included in the main paper for better understanding.
The second equation under line 106: k appears in the left hand side but not in the right hand side.

---------------------

After rebuttal:

The new ablation study results partially answer my concerns, so I will raise my rating to 6.

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a multilingual pretrained language model that learns from universal dependency parsing as well as language modeling. The UD layer is done on top of the transformer by following a biaffine dependency parsing (Dozat and Manning.) and the language modeling is learned with a MLM layer as in BERT/RoBERTa.  For MLM they use 104 languages while for UD learning they use 60 languages present in the Universal Dependencies project. The evaluation is done in XNLI, XCuad, UD parsing, SPMRL constituency parsing (SPMRL was a workshop on parsing for morphologically rich languages, in this case they use the constituent version of those), PTB and CTB



main_review: This work is similar to Swayamdipta et al. 2019 https://arxiv.org/pdf/1908.11047.pdf. In contrast to the findings in Swayamdipta et al. 2019, their model outperforms multilingual-BERT and XLM-R, on cross-lingual natural language understanding benchmarks and parsing tasks.

The evaluation is done in XNLI, XCuad and UD parsing. The UD parsing evaluation is definitely boosted given that the pretraining has seen that data.
Line 196: why not separating SPMRL as another dataset?

Line 25:  "some work has shown that, m-BERT does not have equally high-quality representation for all of the languages it includes and for some languages, m-BERT performs worse than non-BERT" --> does your work address this issue?

- “multilingual PrLMs only focus on plain text and ignore obvious 6 universal linguistic structure clues” --> There is a body of work showing that multilingual pretrained LMs are able to learn syntax without supervision.  (Hewitt and Manning. NAACL 2019) (Perez-Mayos et al. EACL 2021) (Maudslay and Cotterell 2021) among others. You should cite this work and contrast with your approach.

Lines 35-36: those are 2015 and 2016 papers that show the effectiveness of parsing in downstream applications. Is there something a bit more recent?

Lines 168-169: how were the number of layers selected?

Figure 1 is not very clear. Also why is it not part of the main paper? Have you tried just sharing the weights between the two tasks instead of having the US integration. This part is not properly motivated from the write up.

Minor comment: The math in the paper is not always properly written (e.g. line 143, that equation seems lost) I find this guide very useful: https://web.cs.ucdavis.edu/~amenta/w10/writingman.pdf

Tables 1, 2, 3 and 4 : some of the results in bold are not the best results. E.g. First column RoBERTa is 91.3 while your model is 89.4. In table 2 some columns do not have result in bold when those are from the model without syntax. For some reason table 4 does not have bold scores.

Tables 1, 2, 3 and 4: I would recommend reporting an average score across 5 models trained on different random seeds and the standard deviation. Many of the results here are very similar to each other, and it is not clear whether there is any statistical significance.

I am glad to see Section 5, but it  would significantly improve by probing the linguistic knowledge of the generated pretrained LM following (Hewitt and Manning .2019 (https://aclanthology.org/N19-1419/), not only the score in downstream applications.
Table 7: why ar and eu? why not other languages?

worth citing: https://arxiv.org/abs/1904.02099

The paper does not explain how they manage to interleave the training between the two main tasks. In one of them they have millions of sentences, in the other (UD) way less. How do you ensure that there is no catastrophic forgetting  or overfitting on the smaller dataset during training?
limitations_and_societal_impact: Minor call-out: The authors are not reporting the total amount of compute and the type of resources used. It is likely that this kind of work cannot be replicated by most academic labs.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method to pretrain large-scale masked language models while integrating syntactic information from parses. Their experiments and analysis show that their method is effective in a range of crosslingual transfer tasks.  The pretraining objective involves predicting dependency parsing in addition to masked language modeling, which is commonly used in many pretrained models. 
main_review: **Strengths**
- This paper presents a straightforward way of integrating parsing information into a pretraining objective. They proposed a staged training strategy that is well motivated to address parsing failure in an early stage. 

**Weaknesses**

**Updates after the author response**
1. I sincerely apologize for missing the XML-R large rows in the tables. All questions except the last one are fully addressed. 
2. Regarding the last question, I agree that making good use of existing UD corpora is a promising and important direction. And as the authors point out in the response, it's very encouraging to see that some languages without parsing data still benefit from their model. This makes me think that it'd be informative to analyze results by breaking down the improvements by whether or not each language is included in the UD datasets (or more thoroughly, the size of the UD corpus for each language to see the effect of the amount of parsing data). 
3. As other reviewers suggest, there is still a concern about the assessment of the effect of parsing pretraining. Ablation studies would be strengthened if they were done on downstream tasks. Moreover, since UD corpora can provide additional pretraining data even without UD annotations, it can be useful to add the text data to the pretraining data for the XLM baselines as well. For example, it's possible that the text data from the specific domain, not just the UD parses, is contributing to the improvement over the XLM baselines (though this is perhaps only critical in UD parsing results). 
4. I updated my score accordingly. 
----

- I have several critical questions for the authors to be addressed. In particular, I have questions regarding their experimental settings and empirical comparisons. Please see them below. 

**Questions for the authors**
- In Table 1, why is there no XLM-R large in the crosslingual setting? The authors make comparisons for English between XLM-R large and the corresponding UD-XLM-R large, but do not compare XLM large in the crosslingual setting. The All-FT results suggest that XLM-R large substantially would outperform XLM in crosslingual transfer as well, but no head-to-head comparison is made between UD-XLM-R large and XLM-R large for crosslingual transfer. 
- And numbers in bold face for UD-XLM-R large are confusing; since the corresponding XLM-R large is not evaluated in the crosslingual setting, their results are not truly comparable to other models with limited capacity. 
- Conversely in Table 1, have you tried your models like UD-XLM-R large in the All-FT setting? Did it perform equally well compared to XML-R large? I wonder if the proposed method can work in high resource settings as well. This would help us understand whether the proposed model is only effective in low-resource settings.
- Table 2 also has the same problem: while numbers from XLM-R large are in bold face, there is no head-to-head comparison. 
- Interestingly, XLM-R and UD-XLM-R are compared in Tables 4-6 for other tasks. This makes one wonder more why it is not done in the other tables. 
- I might be missing this, but what (labeled) data is used for $L_{USL}$ training (line 4 in Algorithm 1)? Did you use the UD corpora? Or did you use automatically parsed data from the UD treebank? The difference in training data is important to understand what is needed for the proposed method and whether comparisons of different models are fair. For example, If UD parsing data is necessary, whether or not it is automatically parsed or parsed by human annotators, the proposed method is not expected to be practical in many truly low-resource languages whose parses/parsers are not readily available. Since the paper emphasizes results from crosslingual transfer, this is an important point. 
 
**Typos and Minor Issues**
- Line 44: "meantime, we propose and structural encoder to encode" -> "meantime, we propose a structural encoder to encode".
- Line 55: "mono-lingual". Please be consistent in hyphenation throughout the paper. 
- Lines 265-266: run-on sentence. 




limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a multilingual language model pre-training strategy which incorporates both universal dependency parsing and masked language modeling objectives. The authors integrate two additional layers, namely Universal Structure Learning layer and Universal Structure Integration, to mBERT and XLM-R for explicitly learning the language structures representations. The experimental results show that the proposed model improves both mBERT and XLM-R in multiple cross-lingual tasks, including Cross-lingual Natural Language Inference (XNLI), Cross-lingual Question Answering (XQuAD), and Universal Linguistic Structure Parsing.

main_review: Originality: 
- Incorporating language structures into language model pre-training has been explored in the monolingual setting[1,2]. This work extends this idea to the multilingual language model pre-training, and shows the improvement in cross-lingual tasks. However, it would be helpful to highlight the differences other than the pre-training data.

Quality: 
- The proposed method is technically sound. The claims are supported by extensive experiments in multiple well-known datasets. To further strengthen the paper, the authors could evaluate their models on XTREME [3].

Clarity: 
- This paper is well organized. However, some training details such as learning rate and training batch size are missing. It is also not cleat how the authors balance the training for USL and MLM as they are trained with different data.

Significance: 
- Learning robust multilingual representation is an important task. This paper achieved strong empirical results in many cross lingual tasks.

Additional Questions (feedback):
- Why both mBERT and XLM-R are trained from scratch? What is the performance if both models are initialized with original weights?
- The ablation study is only conducted on a linguistic structure parsing dataset (SPMRL), and the authors reach the conclusion that "the additional parser structure, even though it is not trained by the UD supervision, can still improve the performance, and the improvement is greater than that of w/o Universal Structure Integration". However, I would like to see the ablation experiments on cross-lingual NLU benchmarks such as XNLI to validate this conclusion.

**Post-rebuttal updates** The authors response has addressed some of my concerns (rating 5 --> 6). However, I expect authors can conduct additional ablation study on XNLI and XQuAD to verify the effectiveness of Universal Structure Integration and UD Pretraining.

[1] StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding. ICLR 2020
[2] LIMIT-BERT : Linguistics Informed Multi-Task BERT. EMNLP 2020
[3] XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. ICML 2020


limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper breaks new ground by adding explicit structural supervision to a multilingual pretrained LM, and looking at the interaction between structural supervision (which is known to be helpful) and multilinguality. Overall the work is carefully done and well motivated, and the results are strong, indicating that “universal” structural information improves multilingual transferability. This is an important and necessary, but not overly surprising, next step. The risk of “obvious” papers like this is that they are done poorly, but this paper does a good job (although the AC would encourage the authors to include the follow-up results on XNLI/XQuAD tasks whenever the paper ends up being published).
Decision: Accept (Poster)