DTAI-KULeuven /robbertje-1-gb-non-shuffled

RobBERTje is a collection of distilled models based on RobBERT. There are multiple models with different sizes and different training settings, which you can choose for your use-case.

We are also continuously working on releasing better-performing models, so watch the repository for updates.

News

• February 21, 2022: Our paper about RobBERTje has been published in volume 11 of CLIN journal!
• July 2, 2021: Publicly released 4 RobBERTje models.
• May 12, 2021: RobBERTje was accepted at CLIN31 for an oral presentation!

The models

Model Description Parameters Training size Huggingface id
Non-shuffled Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation. 74 M 1 GB this model
Shuffled Trained on the publicly available and shuffled OSCAR corpus. 74 M 1 GB DTAI-KULeuven/robbertje-1-gb-shuffled
Merged (p=0.5) Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%. 74 M 1 GB DTAI-KULeuven/robbertje-1-gb-merged
BORT A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT). 46 M 1 GB DTAI-KULeuven/robbertje-1-gb-bort

Results

Intrinsic results

We calculated the pseudo perplexity (PPPL) from cite, which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.

Model PPPL
RobBERT (teacher) 7.76
Non-shuffled 12.95
Shuffled 18.74
Merged (p=0.5) 17.10
BORT 26.44

Extrinsic results

We also evaluated our models on sereral downstream tasks, just like the teacher model RobBERT. Since that evaluation, a Dutch NLI task named SICK-NL was also released and we evaluated our models with it as well.

Model DBRD DIE-DAT NER POS SICK-NL
RobBERT (teacher) 94.4 99.2 89.1 96.4 84.2
Non-shuffled 90.2 98.4 82.9 95.5 83.4
Shuffled 92.5 98.2 82.7 95.6 83.4
Merged (p=0.5) 92.9 96.5 81.8 95.2 82.8
BORT 89.6 92.2 79.7 94.3 81.0
Mask token: <mask>