Implement Roberta PreLayerNorm
Author: AndreasMadsen
Pull request: yes

# What does this PR do?

This PR implements Roberta PreLayerNorm as used in https://arxiv.org/abs/2202.08005, code provided at https://github.com/princeton-nlp/DinkyTrain. The model is equivariant to using the `--encoder-normalize-before` flag in fairseq. The addition of this model was discussed in https://github.com/huggingface/transformers/issues/19877.

Note that checkpoints provided by https://arxiv.org/abs/2202.08005, such as https://huggingface.co/princeton-nlp/efficient_mlm_m0.40 are not valid as they assume an hacked RoBERTa model provided in https://github.com/princeton-nlp/DinkyTrain. Additionally, the checkpoints contain extra weights which are never used due to [a bug](https://github.com/princeton-nlp/DinkyTrain/blob/main/huggingface/modeling_roberta_prelayernorm.py#L185) in their convention code. The conversion script provided in this PR fixes those issues. I'm not sure what the appropriate migration is for potentially fixing those checkpoints. For now, I have uploaded the corrected checkpoints to https://huggingface.co/andreasmadsen/efficient_mlm_m0.40.

Fixes #19877

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

/cc @sgugger who commented on the original issue https://github.com/huggingface/transformers/issues/19877


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @ArthurZucker, thanks for looking at this. I fixed minor issues reported by the GitHub testing runners. However, the CircleCI job appears to have some configuration issues that I don't think I'm responsible for. Any insight would be appreciated :)

While we're waiting for @ArthurZucker to come back from vacation next week, could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

@sgugger Thanks for the tip! I refreshed the OAuth and rebased the branch.

@sgugger thanks for the help. I fixed the remaining issues, all tests appear to be passing now.

I fixed everything that was mentioned. However, I couldn't remember/find what command to run to update the documentation files. I assume a test will fail and tell me.

Lol, there appears to be some issue with a Linux distribution repository. I will make a force push later to refresh the CI. 

Thanks for the fixes! Could you also resolve the merge conflicts! Will then ask for a final review from @sgugger 

@ArthurZucker I rebased it. However, some of the README diffs contain some space ` ` changes. If you want me to fix that please let me know the `make` command to update the README files. 

Normally `make fixup` or just `make style` should do the trick 😉 

Let's just remove the code to have the correct slice in the different tests and we can merge!

@ArthurZucker Thanks, I appreciate it.

@ArthurZucker As I mentioned before, similar comments were included in the RoBERTa testfile which is why I kept them. However, I have removed them now as you suggested.

Appears to be an unrelated error: 

```
FAILED examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_swag_no_trainer - AssertionError: 0.7 not greater than or equal to 0.8
```

I will try to rebase.

Cool thanks for the addition!  🚀