Add ELECTRA/DeBERTa v3 pretraining script (replaced token detection pretraining)
Author: BramVanroy
Pull request: no

### Feature request

It would be welcome to add a pretraining script for the replaced token detection task that [ELECTRA](https://github.com/google-research/electra) and, later, [DeBERTa v3](https://github.com/microsoft/DeBERTa/tree/master/experiments/language_model#pre-training-with-replaced-token-detection-task) were trained on.

Note that DeBERTa v3 models are actually of type [DeBERTa v2](https://huggingface.co/docs/transformers/model_doc/deberta-v2) under the hood according to [the config file](https://huggingface.co/microsoft/deberta-v3-large/blob/main/config.json#L2).

### Motivation

While DeBERTa v3 and especially ELECTRA are quite "old" in terms of LLM life spans, for completeness' sake it could be worthwhile to add a training example that is fully compatible with all the recent developments in the HF eco system (accelerate, peft, datasets, evaluate etc.).

### Your contribution

Depending on the interest and my own time I can either review or contribute to this as well.

Comments:
Hey @BramVanroy really good idea!

I think a good start would be the codebasis of latest CamemBERTa model. It uses own DeBERTa v3 pretraining code  (modified from the ELECTRA implementation from NVIDIA). In general, DeBERTa v3 uses Gradient-Disentangled Embedding Sharing (GDES) in pretraining compared to v2, which is also implemented in CamemBERTa repository.

Repo is here: https://github.com/WissamAntoun/CamemBERTa

@WissamAntoun is the first author of CamemBERTa paper and also active here :hugs: 

Great find @stefan-it! I see that the code is modified from the [NVIDIA repo](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/LanguageModeling/ELECTRA). That's probably a great starting point. Personally, I'd also like to see a `torch` equivalent (which I might work on if no one else picks this up).

Hey @BramVanroy ,

regarding to ELECTRA and PyTorch I recently discovered this repo:

https://github.com/ficstamas/charmen-electra

It implements a kind of Charformer with ELECTRA, but ELECTRA pretraining is also supported. This could be also a good start and interesting for a PyTorch reference, I'm currently testing the ELECTRA Charformer approach :)

(/cc @ficstamas who is maintainer of that repo :hugs: )

Hey,

@stefan-it Thanks for the cc!

There is an [unofficial](https://github.com/richarddwang/electra_pytorch/tree/master) implementation of ELECTRA which can be a good starting point for you. I used this repository as a reference to make my own. 

Also here is a more documented, stripped down version of [my implementation*](https://gist.github.com/ficstamas/263435c924abdd7f742d9925ab12b0d1) if you need it.

*In this example, I initialized it from a checkpoint, but you can initialize it however you like.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Are there no plans to add official RTD/Deberta-V3 pretraining support?