SlowGuess/ACL_2020_OCR huggingface dataset:
CamembERT: a Tasty French Language Model
Louis Martin\*1,2,3 Benjamin Muller\*2,3 Pedro Javier Ortiz Suarez\*2,3
Yoann Dupont<sup>3</sup> Laurent Romary<sup>2</sup> Éric Villemonde de la Clergerie<sup>2</sup> Djamé Seddah<sup>2</sup> Benoit Sagot<sup>2</sup>
$^{1}$ Facebook AI Research, Paris, France  $^{2}$ Inria, Paris, France
$^{3}$ Sorbonne Université, Paris, France
louismartin@fb.com
{benjamin.muller, pedro.ortiz, laurent.romary,
eric.de_ta_clergerie, djame.seddah, benoit.sagot}@inria.fr
yoa.dupont@gmail.com
Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models—in all languages except English—very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets  $(130 + \mathrm{GB})$ . Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.
Pretrained word representations have a long history in Natural Language Processing (NLP), from noncontextual (Brown et al., 1992; Ando and Zhang, 2005; Mikolov et al., 2013; Pennington et al., 2014) to contextual word embeddings (Peters et al., 2018; Akbik et al., 2018). Word representations are usually obtained by training language model architectures on large amounts of textual data and then fed as an input to more complex task-specific architectures. More recently, these specialized architectures have been replaced altogether by large-scale pretrained language models which are fine-tuned for each application considered. This shift has resulted in large improvements in performance over a wide
range of tasks (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019; Raffel et al., 2019).
These transfer learning methods exhibit clear advantages over more traditional task-specific approaches. In particular, they can be trained in an unsupervised manner, thereby taking advantage of the information contained in large amounts of raw text. Yet they come with implementation challenges, namely the amount of data and computational resources needed for pretraining, which can reach hundreds of gigabytes of text and require hundreds of GPUs (Yang et al., 2019; Liu et al., 2019). This has limited the availability of these state-of-the-art models to the English language, at least in the monolingual setting. This is particularly inconvenient as it hinders their practical use in NLP systems. It also prevents us from investigating their language modelling capacity, for instance in the case of morphologically rich languages.
Although multilingual models give remarkable results, they are often larger, and their results, as we will observe for French, can lag behind their monolingual counterparts for high-resource languages.
In order to reproduce and validate results that have so far only been obtained for English, we take advantage of the newly available multilingual corpora OSCAR (Ortiz Suárez et al., 2019) to train a monolingual language model for French, dubbed CamemBERT. We also train alternative versions of CamemBERT on different smaller corpora with different levels of homogeneity in genre and style in order to assess the impact of these parameters on downstream task performance. CamemBERT uses the RoBERTa architecture (Liu et al., 2019), an improved variant of the high-performing and widely used BERT architecture (Devlin et al., 2019).
We evaluate our model on four different downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER) and natural language inference (NLI).
* Equal contribution. Order determined alphabetically.
7203
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203-7219 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
CamemBERT improves on the state of the art in all four tasks compared to previous monolingual and multilingual approaches including mBERT, XLM and XLM-R, which confirms the effectiveness of large pretrained language models for French.
- First release of a monolingual RoBERTa model for the French language using recently introduced large-scale open source corpora from the Oscar collection and first outside the original BERT authors to release such a large model for an other language than English.
- We achieve state-of-the-art results on four downstream tasks: POS tagging, dependency parsing, NER and NLI, confirming the effectiveness of BERT-based language models for French.
- We demonstrate that small and diverse training sets can achieve similar performance to large-scale corpora, by analysing the importance of the pretraining corpus in terms of size and domain.
2 Previous work
2.1 Contextual Language Models
From non-contextual to contextual word embeddings The first neural word vector representations were non-contextualized word embeddings, most notably word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014) and fastText (Mikolov et al., 2018), which were designed to be used as input to task-specific neural architectures. Contextualized word representations such as ELMo (Peters et al., 2018) and flair (Akbik et al., 2018), improved the representational power of word embeddings by taking context into account. Among other reasons, they improved the performance of models on many tasks by handling words polysemy. This paved the way for larger contextualized models that replaced downstream architectures altogether in most tasks. Trained with language modeling objectives, these approaches range from LSTM-based architectures such as (Dai and Le, 2015), to the successful transformer-based architectures such as GPT2 (Radford et al., 2019), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and more recently ALBERT (Lan et al., 2019) and T5 (Raffel et al., 2019).
Non-English contextualized models Following the success of large pretrained language models, they were extended to the multilingual setting with multilingual BERT (hereafter mBERT) (Devlin et al., 2018), a single multilingual model for 104 different languages trained on Wikipedia data, and later XLM (Lample and Conneau, 2019), which significantly improved unsupervised machine translation. More recently XLM-R (Conneau et al., 2019), extended XLM by training on 2.5TB of data and outperformed previous scores on multilingual benchmarks. They show that multilingual models can obtain results competitive with monolingual models by leveraging higher quality data from other languages on specific downstream tasks.
A few non-English monolingual models have been released: ELMo models for Japanese, Portuguese, German and Basque² and BERT for Simplified and Traditional Chinese (Devlin et al., 2018) and German (Chan et al., 2019).
However, to the best of our knowledge, no particular effort has been made toward training models for languages other than English at a scale similar to the latest English models (e.g. RoBERTa trained on more than 100GB of data).
BERT and RoBERTa Our approach is based on RoBERTa (Liu et al., 2019) which itself is based on BERT (Devlin et al., 2019). BERT is a multi-layer bidirectional Transformer encoder trained with a masked language modeling (MLM) objective, inspired by the Cloze task (Taylor, 1953). It comes in two sizes: the  $\mathrm{BERT}_{\mathrm{BASE}}$  architecture and the  $\mathrm{BERT}_{\mathrm{LARGE}}$  architecture. The  $\mathrm{BERT}_{\mathrm{BASE}}$  architecture is 3 times smaller and therefore faster and easier to use while  $\mathrm{BERT}_{\mathrm{LARGE}}$  achieves increased performance on downstream tasks. RoBERTa improves the original implementation of BERT by identifying key design choices for better performance, using dynamic masking, removing the next sentence prediction task, training with larger batches, on more data, and for longer.
3 Downstream evaluation tasks
In this section, we present the four downstream tasks that we use to evaluate CamemBERT, namely: Part-Of-Speech (POS) tagging, dependency parsing, Named Entity Recognition (NER) and Natural Language Inference (NLI). We also present the baselines that we will use for comparison.
1Released at: https://camembert-model.fr under the MIT open-source license.
$^{2}$ https://allenlp.org/elmo
7204
Tasks POS tagging is a low-level syntactic task, which consists in assigning to each word its corresponding grammatical category. Dependency parsing consists in predicting the labeled syntactic tree in order to capture the syntactic relations between words.
For both of these tasks we run our experiments using the Universal Dependencies  $(\mathrm{UD})^3$  framework and its corresponding UD POS tag set (Petrov et al., 2012) and UD treebank collection (Nivre et al., 2018), which was used for the CoNLL 2018 shared task (Seker et al., 2018). We perform our evaluations on the four freely available French UD treebanks in UD v2.2: GSD (McDonald et al., 2013), Sequoia4 (Candito and Seddah, 2012; Candito et al., 2014), Spoken (Lacheret et al., 2014; Bawden et al., 2014) $^5$ , and ParTUT (Sanguinetti and Bosco, 2015). A brief overview of the size and content of each treebank can be found in Table 1.
Treebank	#Tokens	#Sentences	Genres
GSD	389,363	16,342	Blogs, News Reviews, Wiki
Sequoia	68,615	3,099	Medical, News Non-fiction, Wiki
Spoken	34,972	2,786	Spoken
ParTUT	27,658	1,020	Legal, News, Wikis
FTB	350,930	27,658	News
Table 1: Statistics on the treebanks used in POS tagging, dependency parsing, and NER (FTB).
We also evaluate our model in NER, which is a sequence labeling task predicting which words refer to real-world objects, such as people, locations, artifacts and organisations. We use the French Treebank $^{6}$  (FTB) (Abeille et al., 2003) in its 2008 version introduced by Candito and Crabbé (2009) and with NER annotations by Sagot et al. (2012). The FTB contains more than 11 thousand entity mentions distributed among 7 different entity types. A brief overview of the FTB can also be found in Table 1.
Finally, we evaluate our model on NLI, using the French part of the XNLI dataset (Conneau et al., 2018). NLI consists in predicting whether a hypothesis sentence is entailed, neutral or contradicts a premise sentence. The XNLI dataset is the exten
sion of the Multi-Genre NLI (MultiNLI) corpus (Williams et al., 2018) to 15 languages by translating the validation and test sets manually into each of those languages. The English training set is machine translated for all languages other than English. The dataset is composed of  $122\mathrm{k}$  train, 2490 development and 5010 test examples for each language. As usual, NLI performance is evaluated using accuracy.
Baselines In dependency parsing and POSTagging we compare our model with:
- mBERT: The multilingual cased version of BERT (see Section 2.1). We fine-tune mBERT on each of the treebanks with an additional layer for POS-tagging and dependency parsing, in the same conditions as our CamembERT model.
-  $XLM_{MLM-TLM}$ : A multilingual pretrained language model from Lample and Conneau (2019), which showed better performance than mBERT on NLI. We use the version available in the Hugging's Face transformer library (Wolf et al., 2019); like mBERT, we fine-tune it in the same conditions as our model.
- UDify (Kondratyuk, 2019): A multitask and multilingual model based on mBERT, UDify is trained simultaneously on 124 different UD treebanks, creating a single POS tagging and dependency parsing model that works across 75 different languages. We report the scores from Kondratyuk (2019) paper.
- UDPipe Future (Straka, 2018): An LSTM-based model ranked  $3^{\text{rd}}$  in dependency parsing and  $6^{\text{th}}$  in POS tagging at the CoNLL 2018 shared task (Seker et al., 2018). We report the scores from Kondratyuk (2019) paper.
- UDPipe Future + mBERT + Flair (Straka et al., 2019): The original UDPipe Future implementation using mBERT and Flair as feature-based contextualized word embeddings. We report the scores from Straka et al. (2019) paper.
In French, no extensive work has been done on NER due to the limited availability of annotated corpora. Thus we compare our model with the only recent available baselines set by Dupont (2017), who trained both CRF (Lafferty et al., 2001) and
<sup>3</sup>https://universaldependencies.org
<sup>4</sup>https://deep-sequoia.inria.fr
5 Speech transcript uncased that includes annotated disfluencies without punctuation
This dataset has only been stored and used on Inria's servers after signing the research-only agreement.
7205
BiLSTM-CRF (Lample et al., 2016) architectures on the FTB and enhanced them using heuristics and pretrained word embeddings. Additionally, as for POS and dependency parsing, we compare our model to a fine-tuned version of mBERT for the NER task.
For XNLI, we provide the scores of mBERT which has been reported for French by Wu and Dredze (2019). We report scores from  $\mathrm{XLM}_{\mathrm{MLM-TLM}}$  (described above), the best model from Lample and Conneau (2019). We also report the results of XLM-R (Conneau et al., 2019).
4 CamemBERT: a French Language Model
In this section, we describe the pretraining data, architecture, training objective and optimisation setup we use for CamemBERT.
Pretrained language models benefits from being trained on large datasets (Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2019). We therefore use the French part of the OSCAR corpus (Ortiz Suárez et al., 2019), a pre-filtered and pre-classified version of Common Crawl.
OSCAR is a set of monolingual corpora extracted from Common Crawl snapshots. It follows the same approach as (Grave et al., 2018) by using a language classification model based on the fastText linear classifier (Grave et al., 2017; Joulin et al., 2016) pretrained on Wikipedia, Tatoeba and SETimes, which supports 176 languages. No other filtering is done. We use a non-shuffled version of the French data, which amounts to 138GB of raw text and 32.7B tokens after subword tokenization.
4.2 Pre-processing
We segment the input text data into subword units using SentencePiece (Kudo and Richardson, 2018). SentencePiece is an extension of Byte-Pair encoding (BPE) (Sennrich et al., 2016) and WordPiece (Kudo, 2018) that does not require pre-tokenization (at the word or token level), thus removing the need for language-specific tokenisers. We use a vocabulary size of  $32k$  subword tokens. These subwords are learned on  $10^{7}$  sentences sampled randomly from the pretraining dataset. We do not use subword regularisation (i.e. sampling from multiple possible segmentations) for the sake of simplicity.
Transformer Similar to RoBERTa and BERT, CamemBERT is a multi-layer bidirectional Transformer (Vaswani et al., 2017). Given the widespread usage of Transformers, we do not describe them here and refer the reader to (Vaswani et al., 2017). CamemBERT uses the original architectures of  $\mathrm{BERT}_{\mathrm{BASE}}$  (12 layers, 768 hidden dimensions, 12 attention heads, 110M parameters) and  $\mathrm{BERT}_{\mathrm{LARGE}}$  (24 layers, 1024 hidden dimensions, 16 attention heads, 335M parameters). CamemBERT is very similar to RoBERTa, the main difference being the use of whole-word masking and the usage of SentencePiece tokenization (Kudo and Richardson, 2018) instead of WordPiece (Schuster and Nakajima, 2012).
Pretraining Objective We train our model on the Masked Language Modeling (MLM) task. Given an input text sequence composed of  $N$  tokens  $x_{1},\ldots ,x_{N}$ , we select  $15\%$  of tokens for possible replacement. Among those selected tokens,  $80\%$  are replaced with the special  $\langle \mathrm{MASK} \rangle$  token,  $10\%$  are left unchanged and  $10\%$  are replaced by a random token. The model is then trained to predict the initial masked tokens using cross-entropy loss.
Following the RoBERTa approach, we dynamically mask tokens instead of fixing them statically for the whole dataset during preprocessing. This improves variability and makes the model more robust when training for multiple epochs.
Since we use SentencePiece to tokenize our corpus, the input tokens to the model are a mix of whole words and subwords. An upgraded version of BERT $^{8}$  and Joshi et al. (2019) have shown that masking whole words instead of individual subwords leads to improved performance. Whole-word Masking (WWM) makes the training task more difficult because the model has to predict a whole word rather than predicting only part of the word given the rest. We train our models using WWM by using whitespaces in the initial untokenized text as word delimiters.
WWM is implemented by first randomly sampling  $15\%$  of the words in the sequence and then considering all subword tokens in each of this  $15\%$  for candidate replacement. This amounts to a proportion of selected tokens that is close to the original  $15\%$ . These tokens are then either replaced by
<sup>7</sup>https://commoncrawl.org/about/
https://github.com/google-research/bert/blob/master/README.md
7206
<MAsk> tokens (80%), left unchanged (10%) or replaced by a random token.
Subsequent work has shown that the next sentence prediction (NSP) task originally used in BERT does not improve downstream task performance (Lample and Conneau, 2019; Liu et al., 2019), thus we also remove it.
Optimisation Following (Liu et al., 2019), we optimize the model using Adam (Kingma and Ba, 2014)  $(\beta_{1} = 0.9, \beta_{2} = 0.98)$  for 100k steps with large batch sizes of 8192 sequences, each sequence containing at most 512 tokens. We enforce each sequence to only contain complete paragraphs (which correspond to lines in the our pretraining dataset).
Pretraining We use the RoBERTa implementation in the fairseq library (Ott et al., 2019). Our learning rate is warmed up for  $10\mathrm{k}$  steps up to a peak value of 0.0007 instead of the original 0.0001 given our large batch size, and then fades to zero with polynomial decay. Unless otherwise specified, our models use the BASE architecture, and are pretrained for  $100\mathrm{k}$  backpropagation steps on 256 Nvidia V100 GPUs (32GB each) for a day. We do not train our models for longer due to practical considerations, even though the performance still seemed to be increasing.
4.4 Using CamemBERT for downstream tasks
We use the pretrained CamemBERT in two ways. In the first one, which we refer to as fine-tuning, we fine-tune the model on a specific task in an end-to-end manner. In the second one, referred to as feature-based embeddings or simply embeddings, we extract frozen contextual embedding vectors from CamemBERT. These two complementary approaches shed light on the quality of the pretrained hidden representations captured by CamemBERT.
Fine-tuning For each task, we append the relevant predictive layer on top of CamemBERT's architecture. Following the work done on BERT (Devlin et al., 2019), for sequence tagging and sequence labeling we append a linear layer that respectively takes as input the last hidden representation of the  $
$  special token and the last hidden representation of the first subword token of each word. For dependency parsing, we plug a bi-affine graph predictor head as inspired by Dozat and Manning (2017). We refer the reader to this article for more details on this module. We fine-tune on XNLI
by adding a classification head composed of one hidden layer with a non-linearity and one linear projection layer, with input dropout for both.
We fine-tune CamemBERT independently for each task and each dataset. We optimize the model using the Adam optimiser (Kingma and Ba, 2014) with a fixed learning rate. We run a grid search on a combination of learning rates and batch sizes. We select the best model on the validation set out of the 30 first epochs. For NLI we use the default hyperparameters provided by the authors of RoBERTa on the MNLI task. Although this might have pushed the performances even further, we do not apply any regularisation techniques such as weight decay, learning rate warm-up or discriminative finetuning, except for NLI. We show that fine-tuning CamemBERT in a straightforward manner leads to state-of-the-art results on all tasks and outperforms the existing BERT-based models in all cases. The POS tagging, dependency parsing, and NER experiments are run using Hugging Face's Transformer library extended to support CamemBERT and dependency parsing (Wolf et al., 2019). The NLI experiments use the faireq library following the RoBERTa implementation.
Embeddings Following Straková et al. (2019) and Straka et al. (2019) for mBERT and the English BERT, we make use of CamemBERT in a feature-based embeddings setting. In order to obtain a representation for a given token, we first compute the average of each sub-word's representations in the last four layers of the Transformer, and then average the resulting sub-word vectors.
We evaluate CamemBERT in the embeddings setting for POS tagging, dependency parsing and NER; using the open-source implementations of Straka et al. (2019) and Straková et al. (2019).<sup>10</sup>
5 Evaluation of CamemBERT
In this section, we measure the performance of our models by evaluating them on the four aforementioned tasks: POS tagging, dependency parsing, NER and NLI.
9More details at https://github.com/pytorch/ fairseq/blob/master/examples/roberta/ README.glue.md.
10UDPipe Future is available at https://github. com/CoNLL-UD-2018/UDPipe-Future, and the code for nested NER is available at https://github.com/ ufal/acl2019_nested_ner.
7207
MODEL	GSD	SEQUOIA	SPOKEN	PARTUT
UPOS	LAS	UPOS	LAS	UPOS	LAS	UPOS	LAS
mBERT (fine-tuned)	97.48	89.73	98.41	91.24	96.02	78.63	97.35	91.37
XLMMLM-TLM (fine-tuned)	98.13	90.03	98.51	91.62	96.18	80.89	97.39	89.43
UDify (Kondratyuk, 2019)	97.83	91.45	97.89	90.05	96.23	80.01	96.12	88.06
UDPipe Future (Straka, 2018)	97.63	88.06	98.79	90.73	95.91	77.53	96.93	89.63
+ mBERT + Flair (emb.) (Straka et al., 2019)	97.98	90.31	99.32	93.81	97.23	81.40	97.64	92.47
CamemBERT (fine-tuned)	98.18	92.57	99.29	94.20	96.99	81.37	97.65	93.43
UDPipe Future + CamemBERT (embeddings)	97.96	90.57	99.25	93.89	97.09	81.81	97.50	92.32
Table 2: POS and dependency parsing scores on 4 French treebanks, reported on test sets assuming gold tokenization and segmentation (best model selected on validation out of 4). Best scores in bold, second best underlined.
Model	F1
SEM (CRF) (Dupont, 2017)	85.02
LSTM-CRF (Dupont, 2017)	85.57
mBERT (fine-tuned)	87.35
CamemBERT (fine-tuned)	89.08
LSTM+CRF+CamemBERT (embeddings)	89.55
Table 3: NER scores on the FTB (best model selected on validation out of 4). Best scores in bold, second best underlined.
Model	Acc.	#Params
mBERT (Devlin et al., 2019)	76.9	175M
XLMMLM-TLM (Lample and Conneau, 2019)	80.2	250M
XLM-RBASE (Conneau et al., 2019)	80.1	270M
CamemBERT (fine-tuned)	82.5	110M
Supplement: LARGE models
XLM-RLARGE (Conneau et al., 2019)	85.2	550M
CamemBERTLARGE (fine-tuned)	85.7	335M
Table 4: NLI accuracy on the French XNLI test set (best model selected on validation out of 10). Best scores in bold, second best underlined.
POS tagging and dependency parsing For POS tagging and dependency parsing, we compare CamemBERT with other models in the two settings: fine-tuning and as feature-based embeddings. We report the results in Table 2.
CamemBERT reaches state-of-the-art scores on all treebanks and metrics in both scenarios. The two approaches achieve similar scores, with a slight advantage for the fine-tuned version of CamemBERT, thus questioning the need for complex task-specific architectures such as UDPipe Future.