metadata
language: es
license: CC-BY 4.0
tags:
- spanish
- roberta
pipeline_tag: fill-mask
widget:
- text: Fui a la librería a comprar un <mask>.
BERTIN
BERTIN is a series of BERT-based models for Spanish. This one is a RoBERTa-large model trained from scratch on the Spanish portion of mC4 using Flax, including training scripts.
This is part of the Flax/Jax Community Week, organised by HuggingFace and TPU usage sponsored by Google.
Spanish mC4
The Spanish portion of mC4 containes about 416 million records and 235 billion words.
$ zcat c4/multilingual/c4-es*.tfrecord*.json.gz | wc -l
416057992
$ zcat c4/multilingual/c4-es*.tfrecord-*.json.gz | jq -r '.text | split(" ") | length' | paste -s -d+ - | bc
235303687795
Team members
- Javier de la Rosa (versae)
- Eduardo González (edugp)
- Paulo Villegas (paulo)
- Pablo González de Prado (Pablogps)
- Manu Romero (mrm8488)
- María Grandury (mariagrandury)