Added test cases for rembert refering to albert and reformer test_tok…
Author: nileshkokane01
Pull request: yes


# What does this PR do?

It addresses #16627 to create test cases for RemBert.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @SaulLu 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @ArthurZucker 

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
I do not know why tests_tf, tests_torch, run_tests and tests_jax is failing. 
Can anyone let me know? I can work on it to resolve.

> make sure the mask_token is initialized the same way as an added token for both the fast and the slow tokenizers

You are right.  [This](https://github.com/huggingface/transformers/blob/main/src/transformers/models/rembert/tokenization_rembert_fast.py#L120) is absent in slow token for rembert.

Commenting that line I am not getting any failed cases. 

How to tackle such test cases for which the mask token are initialized differently?
Should the constructor be passed with a appropriate mask_token?



` self.assertDictEqual(EXPECTED_ADDED_TOKENS_DECODER, tokenizer_fasadded_tokens_decoder)
E                           AssertionError: {0: A[454 chars]trip=False, single_word=False, noalized=Fals[123 chars]rue)} != {0: A[454 chars]trip=True, single_word=False, normalized=False22 chars]rue)}
E                           Diff is 870 characters long. Set self.maxDiff to None to see it.`

I overrided test_added_tokens_serialization to clear the following error. 
Do not know if its a right way. 


`
  AssertionError: {0: A[454 chars]trip=False, single_word=False, normalized=Fals[123 chars]rue)} != {0: A[454 chars]trip=True, single_word=False, normalized=False[122 chars]rue)}E                           Diff is 870 characters long. Set self.maxDiff to None to see it.

tests\test_tokenization_common.py:4128: AssertionError


`



It's usually a bit tricky!

Hi @ArthurZucker ,

Thanks for your suggestion. 

I dig in more deeper and found out that  [this](https://github.com/huggingface/transformers/blob/main/tests/test_tokenization_common.py#L4084) line for slow tokens is having AddedToken for [MASK] with lstrip= False. 

However,  [ this](https://github.com/huggingface/transformers/blob/main/tests/test_tokenization_common.py#L4123) will give the instance of Fast Token with  lstrip = True as it invokes the constructor  RemBertTokenizerFast where lstrip = True explicitly. 

The above issue can be fixed and all test Pass- by setting [this](https://github.com/huggingface/transformers/blob/main/src/transformers/models/rembert/tokenization_rembert_fast.py#L120)  **lstrip= False**  and that requires change in file outside test_tokenization_rembert.py , however , the comment above that line gives the reason as to why we have lstrip = True. 

>  Mask token behave like a normal word, i.e. include the space before it

Do let me know, I can send the changes accordingly.

Hey! No, I'm pretty sure this should be fixed by setting the `mask_token` using :

```python
        # Mask token behave like a normal word, i.e. include the space before it
        mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token
```
in the slow tokenizer. Did you try that? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27637). All of your documentation changes will be reflected on that endpoint.

That isn't fixing the issue because it requires  mask_token to be string.

When you call [from_pretrained](https://github.com/huggingface/transformers/blob/main/tests/test_tokenization_common.py#L4083)  all the 'added_token_decoder' gets converted to AddedToken instance from [here](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L2183) based on the config file .

Therefore, the slow token will have lstrip as False, despite the line added as it checks for instance as string.

No. mask_token does not need to be a string it has to be an AddedToken, otherwise it's casted to addedToken when adding it to the vocab if it's not part of the vocab. Anyway what I am saying is that this test works for the ~100 or so tokenizers, to skip it we need a good reason like a very edgy case or whatever,  otherwise it means that loading the tokenizers from slow to fast, from the hub etc does not produce correct results. 

🤗 

Hi,

> No. mask_token does not need to be a string it has to be an AddedToken

If this is the case, the following line will fix the issue. I just verified it. 

` # Mask token behave like a normal word, i.e. include the space before it
   mask_token = AddedToken('[MASK]', lstrip=True, rstrip=False, normalized=False) `

If the above change to slow token is okay, I'll send a patch, otherwise we have to rely on 'finding' a strong case to skip the the slow->fast test.

Yeah the above change should be alright mostly if all tests pass! 

I have made the necessary changes. Can you have a look ? 

can you Pl check the latest commit.

@ArthurZucker 

Thanks for your support!