Rziane/flaubert-base-kreyol-RH

Continued MLM pre-training of flaubert/flaubert_base_cased on a custom text corpus.

Training

• base model: flaubert/flaubert_base_cased
• epochs: 8.0
• batch size (per device): 64
• gradient accumulation: 1
• learning rate: 5e-05
• scheduler: cosine (warmup 0.06)
• weight decay: 0.01
• max length: 128
• mlm probability: 0.15
• precision: fp16
• seed: 42

Final eval

• eval_loss: 1.2977
• eval_runtime: 295.1399
• eval_samples_per_second: 804.4630
• eval_steps_per_second: 6.2850
• epoch: 8.0000
• perplexity: 3.6607

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained('Rziane/flaubert-base-kreyol-RH')
model = AutoModelForMaskedLM.from_pretrained('Rziane/flaubert-base-kreyol-RH')