|
--- |
|
language: |
|
- es |
|
thumbnail: "url to a thumbnail used in social sharing" |
|
license: apache-2.0 |
|
datasets: |
|
- oscar |
|
--- |
|
|
|
# SELECTRA: A Spanish ELECTRA |
|
|
|
SELECTRA is a Spanish pre-trained language model based on [ELECTRA](https://github.com/google-research/electra). |
|
We release a `small` and `medium` version with the following configuration: |
|
|
|
| Model | Layers | Embedding/Hidden Size | Params | Vocab Size | Max Sequence Length | Cased | |
|
| --- | --- | --- | --- | --- | --- | --- | |
|
| **SELECTRA small** | **12** | **256** | **22M** | **50k** | **512** | **True** | |
|
| [SELECTRA medium](https://huggingface.co/Recognai/selectra_medium) | 12 | 384 | 41M | 50k | 512 | True | |
|
|
|
**SELECTRA small (medium) is about 5 (3) times smaller than BETO but achieves comparable results** (see Metrics section below). |
|
|
|
## Usage |
|
|
|
From the original [ELECTRA model card](https://huggingface.co/google/electra-small-discriminator): "ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN." |
|
The discriminator should therefore activate the logit corresponding to the fake input token, as the following example demonstrates: |
|
|
|
```python |
|
from transformers import ElectraForPreTraining, ElectraTokenizerFast |
|
|
|
discriminator = ElectraForPreTraining.from_pretrained("Recognai/selectra_small") |
|
tokenizer = ElectraTokenizerFast.from_pretrained("Recognai/selectra_small") |
|
|
|
sentence_with_fake_token = "Estamos desayunando pan rosa con tomate y aceite de oliva." |
|
|
|
inputs = tokenizer.encode(sentence_with_fake_token, return_tensors="pt") |
|
logits = discriminator(inputs).logits.tolist()[0] |
|
|
|
print("\t".join(tokenizer.tokenize(sentence_with_fake_token))) |
|
print("\t".join(map(lambda x: str(x)[:4], logits[1:-1]))) |
|
"""Output: |
|
Estamos desayun ##ando pan rosa con tomate y aceite de oliva . |
|
-3.1 -3.6 -6.9 -3.0 0.19 -4.5 -3.3 -5.1 -5.7 -7.7 -4.4 -4.2 |
|
""" |
|
``` |
|
|
|
However, you probably want to use this model to fine-tune it on a downstream task. |
|
We provide models fine-tuned on the [XNLI dataset](https://huggingface.co/datasets/xnli), which can be used together with the zero-shot classification pipeline: |
|
|
|
- [Zero-shot SELECTRA small](https://huggingface.co/Recognai/zeroshot_selectra_small) |
|
- [Zero-shot SELECTRA medium](https://huggingface.co/Recognai/zeroshot_selectra_medium) |
|
|
|
## Metrics |
|
|
|
We fine-tune our models on 4 different down-stream tasks: |
|
|
|
- [XNLI](https://huggingface.co/datasets/xnli) |
|
- [PAWS-X](https://huggingface.co/datasets/paws-x) |
|
- [CoNLL2002 - POS](https://huggingface.co/datasets/conll2002) |
|
- [CoNLL2002 - NER](https://huggingface.co/datasets/conll2002) |
|
|
|
For each task, we conduct 5 trials and state the mean and standard deviation of the metrics in the table below. |
|
To compare our results to other Spanish language models, we provide the same metrics taken from [Table 4](https://huggingface.co/bertin-project/bertin-roberta-base-spanish#results) of the Bertin-project model card. |
|
|
|
| Model | [CoNLL2002](https://huggingface.co/datasets/conll2002) - POS (acc) | [CoNLL2002](https://huggingface.co/datasets/conll2002) - NER (f1) | [PAWS-X](https://huggingface.co/datasets/paws-x) (acc) | [XNLI](https://huggingface.co/datasets/xnli) (acc) | Params | |
|
| --- | --- | --- | --- | --- | --- | |
|
| SELECTRA small | 0.9653 +- 0.0007 | 0.863 +- 0.004 | 0.896 +- 0.002 | 0.784 +- 0.002 | **22M** | |
|
| SELECTRA medium | 0.9677 +- 0.0004 | 0.870 +- 0.003 | 0.896 +- 0.002 | **0.804 +- 0.002** | 41M | |
|
| [mBERT](https://huggingface.co/bert-base-multilingual-cased) | 0.9689 | 0.8616 | 0.8895 | 0.7606 | 178M | |
|
| [BETO](https://huggingface.co/dccuchile/bert-base-spanish-wwm-cased) | 0.9693 | 0.8596 | 0.8720 | 0.8012 | 110M | |
|
| [BSC-BNE](https://huggingface.co/BSC-TeMU/roberta-base-bne) | **0.9706** | **0.8764** | 0.8815 | 0.7771 | 125M | |
|
| [Bertin](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1-512) | 0.9697 | 0.8707 | **0.8965** | 0.7843 | 125M | |
|
|
|
Some details of our fine-tuning runs: |
|
- epochs: 5 |
|
- batch-size: 32 |
|
- learning rate: 1e-4 |
|
- warmup proportion: 0.1 |
|
- linear learning rate decay |
|
- layerwise learning rate decay |
|
|
|
For all the details, check out our [selectra repo](https://github.com/recognai/selectra). |
|
|
|
## Training |
|
|
|
We pre-trained our SELECTRA models on the Spanish portion of the [Oscar](https://huggingface.co/datasets/oscar) dataset, which is about 150GB in size. |
|
Each model version is trained for 300k steps, with a warm restart of the learning rate after the first 150k steps. |
|
Some details of the training: |
|
- steps: 300k |
|
- batch-size: 128 |
|
- learning rate: 5e-4 |
|
- warmup steps: 10k |
|
- linear learning rate decay |
|
- TPU cores: 8 (v2-8) |
|
|
|
For all details, check out our [selectra repo](https://github.com/recognai/selectra). |
|
|
|
**Note:** Due to a misconfiguration in the pre-training scripts the embeddings of the vocabulary containing an accent were not optimized. If you fine-tune this model on a down-stream task, you might consider using a tokenizer that does not strip the accents: |
|
```python |
|
tokenizer = ElectraTokenizerFast.from_pretrained("Recognai/selectra_small", strip_accents=False) |
|
``` |
|
|
|
## Motivation |
|
|
|
Despite the abundance of excellent Spanish language models (BETO, BSC-BNE, Bertin, ELECTRICIDAD, etc.), we felt there was still a lack of distilled or compact Spanish language models and a lack of comparing those to their bigger siblings. |
|
|
|
## Acknowledgment |
|
|
|
This research was supported by the Google TPU Research Cloud (TRC) program. |
|
|
|
## Authors |
|
|
|
- David Fidalgo ([GitHub](https://github.com/dcfidalgo)) |
|
- Javier Lopez ([GitHub](https://github.com/javispp)) |
|
- Daniel Vila ([GitHub](https://github.com/dvsrepo)) |
|
- Francisco Aranda ([GitHub](https://github.com/frascuchon)) |