Add Ernie-M Model to huggingface
Author: susnato
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Ports Ernie-M from paddle to huggingface(pytorch) and also Fixes #21123
I have uploaded the pytorch converted weights [here](https://huggingface.co/susnato/ernie-m-base_pytorch) and [here](https://huggingface.co/susnato/ernie-m-large_pytorch). The paddle2pytorch weights conversion script has been provided there too. 

Work done till now - 
1. ported the weights.
2. Added `configuration_ernie_m.py`
    `from transformers import AutoConfig`
    `config = AutoConfig.from_pretrained("susnato/ernie-m-base_pytorch")`)
3. Added `tokenization_ernie_m.py` (Only Slow Tokenizer implemented)
    `from transformers import ErnieMTokenizer`
    `tokenizer = ErnieMTokenizer.from_pretrained("susnato/ernie-m-base_pytorch")`

4. ErnieMModel in now working.
    `from transformers import AutoModel`
    `model = AutoModel.from_pretrained("susnato/ernie-m-base_pytorch") # susnato/ernie-m-large_pytorch`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [link here](https://github.com/huggingface/transformers/issues/21123)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker and @younesbelkada

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts and @NielsRogge
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
Great work @susnato ! Looking forward to reviewing your PR :) 
Let us know when you think the PR is ready

Hi @younesbelkada the official paddlenlp implementation of ErnieM does not have any LM head class, since it was neither trained on Causal nor on Masked LM. It was pretrained on both Cross Attention Masked LM and Back Translation Masked LM (both implementations are missing in paddlenlp). Do I need to add MaskedLM in this huggingface implementation since it's a encoder based model or should I bypass it and don't include any LM head like the paddlenlp implementation did?

Hi @susnato !
Thanks for your your message
I think this quite depends on the use case of your model. I'd expect most of the users will rely on `ErnieMModel` since it's the model that is present at `paddlepaddle`. If there is an interest to add these models in the future we can always open follow-up PRs


> Hi @susnato ! Thanks for your your message I think this quite depends on the use case of your model. I'd expect most of the users will rely on `ErnieMModel` since it's the model that is present at `paddlepaddle`. If there is an interest to add these models in the future we can always open follow-up PRs

@younesbelkada  Ok, then I will not add any LMhead for now, and also the rest of the model is ready(with all tests passed), I am currently looking why circleci tests are failing.

Thanks! 
Currently some tests are not passing because you need to define a `ERNIE_M_PRETRAINED_CONFIG_ARCHIVE_MAP` inside`configuration_ernie_m.py`, check here how it is done for `bert`: https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/configuration_bert.py

Hi @younesbelkada I added that and did bunch of others changes with make repo-consistency, make style, but when I run make fixup still it says this error
`
python utils/check_config_docstrings.py
Traceback (most recent call last):
  File "/home/susnato/temp_files/transformers/utils/check_config_docstrings.py", line 89, in <module>
    check_config_docstrings_have_checkpoints()
  File "/home/susnato/temp_files/transformers/utils/check_config_docstrings.py", line 85, in check_config_docstrings_have_checkpoints
    raise ValueError(f"The following configurations don't contain any valid checkpoint:\n{message}")
ValueError: The following configurations don't contain any valid checkpoint:
ErnieMConfig
 `


The values I set are  - `ERNIE_M_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "ernie-m-base_pytorch": "https://huggingface.co/susnato/ernie-m-base_pytorch/blob/main/config.json",
    "ernie-m-large_pytorch": "https://huggingface.co/susnato/ernie-m-large_pytorch/blob/main/config.json",
}`

_The documentation is not available anymore as the PR was closed or merged._

Hi @younesbelkada all checks are successful! the PR is ready to review, please review it.

Hi, @younesbelkada I made all the changes that you requested,
Let me know if any other changes are needed or not and if I have missed any!

Hi, @younesbelkada I made all the changes as you requested. The tests are now all successful! Please check it.

Hi @younesbelkada, I made all those changes that you requested.

Thanks a lot @susnato ! Again great work on the integration so far!
Please wait for the next reviewers to add their review and we should be good merging the PR ;) 

Hi @sgugger I made those changes as you requested and the tests are passed too, please review them.

Hi @sgugger please forgive me for the previous unaddressed changes, but now I have tried to make all changes that you addressed. Please check it and let me know if I need to make new changes or not.

Hi, @ArthurZucker I made the changes you said, added the modeling & config file to the `documentation_tests.txt` and also created a new file `tests/models/ernie_m/test_tokenization_ernie_m.py` for testing `ernie-m` tokenizer. 

Also there is one thing I want to mention here regarding tokenizer is that, I saw some inconsistency in ErnieMTokenizer(paddle implementation) regarding how it treats white space, for example - 

`from paddlenlp.transformers import AutoTokenizer`
`tokenizer = AutoTokenizer.from_pretrained("PaddlePaddle/ernie-m-base", from_hf_hub=True)`
`tokenizer.tokenize("The quick brown fox jumps over the lazy dog.")`
```
['▁The', '▁quick', '▁brown', 'fox', '▁jump', 's', '▁over', '▁the', '▁la', 'zy', '▁dog', '.']
```

here, despite "brown fox" being two different words, the tokenizer doesn't seperate them(by "▁") so when we decode them we get 
``
[CLS] The quick brownfox jumps over the lazy dog.[SEP]
``

where there should be a space between brown and fox. This issue is not present in most of the words in vocab.

I managed to solved this issue by inserting a "▁" (at line 215 of `src/transformers/models/ernie_m/tokenization_ernie_m.py`)when we see this condition, but this might lead to very slightly different word_embeddings at times(at most there will be this "▁" character in some sentences between words)! 

Hi @ArthurZucker there seems to be a problem with `tests_tf` of 
```
FAILED tests/models/hubert/test_modeling_tf_hubert.py::TFHubertModelTest::test_dataset_conversion
```
which I think is unrelated to this PR, I rebased to `upstream/main` multiple times but still facing this same issue, could you please have a look at this issue? 

EDIT : this [PR](https://github.com/huggingface/transformers/pull/21606) states this very issue, I will rebase again after it is merged .

Hi @ArthurZucker since the `tests_tf` are failing as before, could you please review this code(the recent changes that I made)? It will be very helpful to me because, then I will be able to work on new suggestions/comments, otherwise I am stuck here until this check is fixed.
I will definitely rebase and push again when the test is fixed but in meantime this PR will gain some progress. :)

Of course! I was waiting for you to ping me 😉 

Hi @ArthurZucker I pushed the changes please check!

Okay! LGTM
@sgugger feel free to merge if you think this is ok! 😉