bertin-project /bertin-base-random-exp-512seqlen

This is a RoBERTa-base model trained from scratch in Spanish.

The training dataset is mc4 subsampling documents to a total of about 50 million examples. Sampling is random. This model continued training from sequence length 128 using 20.000 steps for length 512.

Mask token: <mask>