Abstract: Deep learning-based language models pretrained on large unannotated text corpora have been demonstrated to allow efficient transfer learning for natural language processing, with recent approaches such as the transformer-based BERT model advancing the state of the art across a variety of tasks. While most work on these models has focused on high-resource languages, in particular English, a number of recent efforts have introduced multilingual models that can be fine-tuned to address tasks in a large number of different languages. However, we still lack a thorough understanding of the capabilities of these models, in particular for lower-resourced languages. In this paper, we focus on Finnish and thoroughly evaluate the multilingual BERT model on a range of tasks, comparing it with a new Finnish BERT model trained from scratch. The new language-specific model is shown to systematically and clearly outperform the multilingual. While the multilingual model largely fails to reach the performance of previously proposed methods, the custom Finnish BERT model establishes new state-of-the-art results on all corpora for all reference tasks: part-of-speech tagging, named entity recognition, and dependency parsing. We release the model and all related resources created for this study with open licenses at this https URL .

Introduction: Transfer learning approaches using deep neural network architectures have recently achieved substantial advances in a range of natural language processing (NLP) tasks ranging from sequence labeling tasks such as part-of-speech (POS) tagging and named entity recognition (NER) BIBREF0 to dependency parsing BIBREF1 and natural language understanding (NLU) tasks BIBREF2. While the great majority of this work has focused primarily on English, a number of studies have also targeted other languages, typically through multilingual models. The BERT model of devlin2018bert has been particularly influential, establishing state-of-the-art results for English for a range of NLU tasks and NER when it was released. For most languages, the only currently available BERT model is the multilingual model (M-BERT) trained on pooled data from 104 languages. While M-BERT has been shown to have a remarkable ability to generalize across languages BIBREF3, several studies have also demonstrated that monolingual BERT models, where available, can notably outperform M-BERT. Such results include the evaluation of the recently released French BERT model BIBREF4, the preliminary results accompanying the release of a German BERT model, and the evaluation of ronnqvist-etal-2019-multilingual comparing M-BERT with English and German monolingual models. In this paper, we study the application of language-specific and multilingual BERT models to Finnish NLP. We introduce a new Finnish BERT model trained from scratch and perform a comprehensive evaluation comparing its performance to M-BERT on established datasets for POS tagging, NER, and dependency parsing as well as a range of diagnostic text classification tasks. The results show that 1) on most tasks the multilingual model does not represent an advance over previous state of the art, indicating that multilingual models may fail to deliver on the promise of deep transfer learning for lower-resourced languages, and 2) the custom Finnish BERT model systematically outperforms the multilingual as well as all previously proposed methods on all benchmark tasks, showing that language-specific deep transfer learning models can provide comparable advances to those reported for much higher-resourced languages.

Question: By how much did the new model outperform multilingual BERT?
Answer: For POS,  improvements for cased BERT are 1.26 2.52  0.5 for TDT,  FTB and PUD datasets respectively.
For NER in-domain test set, improvement is  2.11 F1 and for NER out-of-domain test set, improvement is  5.32 F1.
For Dependency parsing, improvements are in range from 3.35 to 6.64 LAS for cased BERT.