---
language: 
  - es
thumbnail: "url to a thumbnail used in social sharing"
tags:
- tag1
- tag2
license: apache-2.0
datasets:
- Oscar
metrics:
- metric1
- metric2
---

# SELECTRA: A Spanish ELECTRA

SELECTRA is a Spanish pre-trained language model based on [ELECTRA](https://github.com/google-research/electra).
We release a `small` and `medium` version with the following configuration:

| Model | Layers | Embedding/Hidden Size | Params | Vocab Size | Max Sequence Length | Cased |
| --- | --- | --- | --- | ---  | --- | --- |
| SELECTRA small | 12 | 256 | 22M | 50k | 512 | True |
| SELECTRA medium | 12 | 384 | 41M | 50k | 512 | True |

## Usage

```python
from transformers import ElectraForPreTraining, ElectraTokenizerFast

discriminator = ElectraForPreTraining.from_pretrained("...")
tokenizer = ElectraTokenizerFast.from_pretrained("...")

sentence_with_fake_token = "Estamos desayunando pan rosa con tomate y aceite de oliva."

inputs = tokenizer.encode(sentence_with_fake_token, return_tensors="pt")
logits = discriminator(inputs).logits.tolist()[0]

print("\t".join(tokenizer.tokenize(sentence_with_fake_token)))
print("\t".join(map(lambda x: str(x)[:4], logits[1:-1])))
"""Output:
Estamos desayun ##ando  pan     rosa    con     tomate  y       aceite  de      oliva   .
-3.1    -3.6    -6.9    -3.0    0.19    -4.5    -3.3    -5.1    -5.7    -7.7    -4.4    -4.2
"""
```

- Links to our zero-shot-classifiers

## Metrics

We fine-tune our models on 4 different down-stream tasks:

 - [XNLI](https://huggingface.co/datasets/xnli)
 - [PAWS-X](https://huggingface.co/datasets/paws-x)
 - [CoNLL2002 - POS](https://huggingface.co/datasets/conll2002)
 - [CoNLL2002 - NER](https://huggingface.co/datasets/conll2002)
 
We provide the mean and standard deviation of 5 fine-tuning runs. 
| Model | 
|
 

## Training  

- Link to our repo

## Motivation

Despite the abundance of excelent Spanish language models (BETO, bertin, etc) we felt there was still a lack of distilled or compact models with comparable metrics to their bigger siblings.