Update README.md
Browse files
README.md
CHANGED
|
@@ -12,6 +12,7 @@ widget:
|
|
| 12 |
- Version 1 (beta): July 15th, 2021
|
| 13 |
- Version 1: July 19th, 2021
|
| 14 |
|
|
|
|
| 15 |
# BERTIN
|
| 16 |
|
| 17 |
BERTIN is a series of BERT-based models for Spanish. The current model hub points to the best of all RoBERTa-base models trained from scratch on the Spanish portion of mC4 using [Flax](https://github.com/google/flax). All code and scripts are included.
|
|
@@ -21,6 +22,7 @@ This is part of the
|
|
| 21 |
|
| 22 |
The aim of this project was to pre-train a RoBERTa-base model from scratch during the Flax/JAX Community Event, in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.
|
| 23 |
|
|
|
|
| 24 |
# Motivation
|
| 25 |
According to [Wikipedia](https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers), Spanish is the second most-spoken language in the world by native speakers (>470 million speakers, only after Chinese, and the fourth including those who speak it as a second language). However, most NLP research is still mainly available in English. Relevant contributions like BERT, XLNet or GPT2 sometimes take years to be available in Spanish and, when they do, it is often via multilanguage versions which are not as performant as the English alternative.
|
| 26 |
|
|
|
|
| 12 |
- Version 1 (beta): July 15th, 2021
|
| 13 |
- Version 1: July 19th, 2021
|
| 14 |
|
| 15 |
+
|
| 16 |
# BERTIN
|
| 17 |
|
| 18 |
BERTIN is a series of BERT-based models for Spanish. The current model hub points to the best of all RoBERTa-base models trained from scratch on the Spanish portion of mC4 using [Flax](https://github.com/google/flax). All code and scripts are included.
|
|
|
|
| 22 |
|
| 23 |
The aim of this project was to pre-train a RoBERTa-base model from scratch during the Flax/JAX Community Event, in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.
|
| 24 |
|
| 25 |
+
|
| 26 |
# Motivation
|
| 27 |
According to [Wikipedia](https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers), Spanish is the second most-spoken language in the world by native speakers (>470 million speakers, only after Chinese, and the fourth including those who speak it as a second language). However, most NLP research is still mainly available in English. Relevant contributions like BERT, XLNet or GPT2 sometimes take years to be available in Spanish and, when they do, it is often via multilanguage versions which are not as performant as the English alternative.
|
| 28 |
|