Edit model card

CamemBERTa: A French language model based on DeBERTa V3

CamemBERTa, a French language model based on DeBERTa V3, which is a DeBerta V2 with ELECTRA style pretraining using the Replaced Token Detection (RTD) objective. RTD uses a generator model, trained using the MLM objective, to replace masked tokens with plausible candidates, and a discriminator model trained to detect which tokens were replaced by the generator. Usually the generator and discriminator share the same embedding matrix, but the authors of DeBERTa V3 propose a new technique to disentagle the gradients of the shared embedding between the generator and discriminator called gradient-disentangled embedding sharing (GDES)

This the first publicly available implementation of DeBERTa V3, and the first publicly DeBERTaV3 model outside of the original Microsoft release.

Preprint Paper: https://inria.hal.science/hal-03963729/

Pre-training Code: https://gitlab.inria.fr/almanach/CamemBERTa

How to use CamemBERTa

Our pretrained weights are available on the HuggingFace model hub, you can load them using the following code:

from transformers import AutoTokenizer, AutoModel, AutoModelForMaskedLM

CamemBERTa = AutoModel.from_pretrained("almanach/camemberta-base")
tokenizer = AutoTokenizer.from_pretrained("almanach/camemberta-base")

CamemBERTa_gen = AutoModelForMaskedLM.from_pretrained("almanach/camemberta-base-generator")
tokenizer_gen = AutoTokenizer.from_pretrained("almanach/camemberta-base-generator")

We also include the TF2 weights including the weights for the model's RTD head for the discriminator, and the MLM head for the generator. CamemBERTa is compatible with most finetuning scripts from the transformers library.

Pretraining Setup

The model was trained on the French subset of the CCNet corpus (the same subset used in CamemBERT and PaGNOL) and is available on the HuggingFace model hub: CamemBERTa and CamemBERTa Generator. To speed up the pre-training experiments, the pre-training was split into two phases; in phase 1, the model is trained with a maximum sequence length of 128 tokens for 10,000 steps with 2,000 warm-up steps and a very large batch size of 67,584. In phase 2, maximum sequence length is increased to the full model capacity of 512 tokens for 3,300 steps with 200 warm-up steps and a batch size of 27,648. The model would have seen 133B tokens compared to 419B tokens for CamemBERT-CCNet which was trained for 100K steps, this represents roughly 30% of CamemBERT’s full training. To have a fair comparison, we trained a RoBERTa model, CamemBERT30%, using the same exact pretraining setup but with the MLM objective.

Pretraining Loss Curves

check the tensorboard logs and plots

Fine-tuning results

Datasets: POS tagging and Dependency Parsing (GSD, Rhapsodie, Sequoia, FSMB), NER (FTB), the FLUE benchmark (XNLI, CLS, PAWS-X), and the French Question Answering Dataset (FQuAD)

CamemBERT (CCNet) 97.59 88.69 89.97 94.62 91.36 81.95 80.98 62.51
CamemBERT (30%) 97.53 87.98 91.04 93.28 88.94 79.89 75.14 56.19
CamemBERTa 97.57 88.55 90.33 94.92 91.67 82.00 81.15 62.01

The following table compares CamemBERTa's performance on XNLI against other models under different training setups, which demonstrates the data efficiency of CamemBERTa.

Model XNLI (Acc.) Training Steps Tokens seen in pre-training Dataset Size in Tokens
mDeBERTa 84.4 500k 2T 2.5T
CamemBERTa 82.0 33k 0.139T 0.319T
XLM-R 81.4 1.5M 6T 2.5T
CamemBERT - CCNet 81.95 100k 0.419T 0.319T

Note: The CamemBERTa training steps was adjusted for a batch size of 8192.


The public model weights are licensed under MIT License. This code is licensed under the Apache License 2.0.


Paper accepted to Findings of ACL 2023.

You can use the preprint citation for now

  TITLE = {{Data-Efficient French Language Modeling with CamemBERTa}},
  AUTHOR = {Antoun, Wissam and Sagot, Beno{\^i}t and Seddah, Djam{\'e}},
  URL = {https://inria.hal.science/hal-03963729},
  NOTE = {working paper or preprint},
  YEAR = {2023},
  MONTH = Jan,
  PDF = {https://inria.hal.science/hal-03963729/file/French_DeBERTa___ACL_2023%20to%20be%20uploaded.pdf},
  HAL_ID = {hal-03963729},
  HAL_VERSION = {v1},


Wissam Antoun: wissam (dot) antoun (at) inria (dot) fr

Benoit Sagot: benoit (dot) sagot (at) inria (dot) fr

Djame Seddah: djame (dot) seddah (at) inria (dot) fr

Downloads last month
Inference API has been turned off for this model.