ELECTRA
Author: LysandreJik
Pull request: yes

Adds ELECTRA to the library.

The script I'm using to compare the different models is this [Github gist](https://gist.github.com/LysandreJik/db4c948f6b4483960de5cbac598ad4ed), coupled to [a modified version of the ELECTRA repository](https://github.com/LysandreJik/electra).

- [x] add model/configuration/tokenization classes
- [x] add conversion scripts
- [x] add tests
- [x] finalize

Let's detail what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-paste or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-paste or create the code for your model in the `modeling_...` files (PyTorch)
- [x] copy-paste or create the code for your model in the `modeling_...` files (TF 2.0)
- [x] copy-paste or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this script to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] add your models and tokenizer to `pipeline.py`
- [x] add a link to your conversion script in the main conversion utility (in `commands/convert.py`)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation itself at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3257?src=pr&el=h1) Report
> Merging [#3257](https://codecov.io/gh/huggingface/transformers/pull/3257?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/012d775b14d1ab673aab7eae151823a74a8525a6&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3257/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3257?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3257   +/-   ##
=======================================
  Coverage   77.54%   77.54%           
=======================================
  Files         103      103           
  Lines       17268    17268           
=======================================
  Hits        13390    13390           
  Misses       3878     3878           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3257?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3257?src=pr&el=footer). Last update [012d775...012d775](https://codecov.io/gh/huggingface/transformers/pull/3257?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Does this add the ability to train a language model using method in ELECTRA? (Thanks!)

It doesn't, we're currently working on a pre-training script incorporating the ELECTRA method but it's still a few weeks out.

> It doesn't, we're currently working on a pre-training script incorporating the ELECTRA method but it's still a few weeks out.

Is this an active branch/issue?  I'm interested in contributing if so, but I can't find it

Not public yet, will let you know when it is @shoarora!