Add regression tests for slow sentencepiece tokenizers. 
Author: PhilipMay
Pull request: yes

This PR adds regression tests for slow sentencepiece tokenizers. These tests are needed for a refactoring in PR #11716 

## Strange findings
- s2t: `_convert_token_to_id` of `"
"` does not give 0

## ToDo
- 
add test for `_convert_token_to_id`
 - done, see `test_convert_token_and_id`
- 
add test for `_convert_id_to_token`
 - done, see `test_convert_token_and_id`
- 
add test for `get_vocab`
 - done
- 
add test for `vocab_size`
 - done
- 
add test for `convert_tokens_to_string`
 - done, see `test_sentencepiece_tokenize_and_convert_tokens_to_string` in `TokenizerTesterMixin` 
- 
add test for pickle
 - is tested in `test_pickle_subword_regularization_tokenizer`
- 
manual review
 - done
- 
fix / add reformer integration test
 - see https://github.com/huggingface/transformers/pull/11737#issuecomment-850769064 - done
- 
add typing
 - done
- 
add docstrings
 - done

Comments:
rebased on master

This PR is ready for review please. @LysandreJik @sgugger 

The failing test is connected to #11731

> Cool, thanks a lot for working on these tests! I think that these are already somewhat covered by the common tests, but they're fast and should help identify issues faster.
> 
> However, in order to make sure PR #11716 can be merged, I was mentioning integration tests, rather than regression/unit tests. For example the ALBERT integration test:
> 
> https://github.com/huggingface/transformers/blob/b8344a274fe13b390fa60c74b76117f5ea8144cb/tests/test_tokenization_albert.py#L108-L152
> 
> Those are particularly important when doing refactors that may affect the encoding/decoding aspect of tokenizers.
> 
> I think this is a bit of a larger work though, so we can post "Good first issues" for the SPM-based tokenizers in a first step so that the community may help.

Ok @LysandreJik .
So I will extend the PR and add integration tests for the `_tokenizer` function like the one you linked above to all sentencepiece tokenizers.

Do you think the already written tests can stay as they are? What other steps are needed?

Hi @PhilipMay, thanks for offering to do it! Feel free to let us know if you would like us to offer some of these to the community, as it can be a bit of work to get every tokenizer tested.

Other than the integration tests, I don't think anything is needed.

Also, you might be interested in rebasing on the `master` branch - we've solved the issue regarding the `run_tests_torch` timing out yesterday so by rebasing you would have reliable CI feedback.

> Hi @PhilipMay, thanks for offering to do it! Feel free to let us know if you would like us to offer some of these to the community, as it can be a bit of work to get every tokenizer tested.

I was thinking to add integration tests for the tokeinzers that I want to refactor (the sentencepiece). And not foll all tokenizers.

**What about this:** In this PR I add integration tests for the **sentencepiece** tokeinzers only - a full list see here #11417 

After that has been merged I (or you) will open an issue asking for similar tests for all tokenizers.

@LysandreJik what do you think?


Yes, sentencepiece tokenizers only, definitely! But even so, that's quite a large number of tokenizers :)

> Also, you might be interested in rebasing on the master branch - we've solved the issue regarding the run_tests_torch timing out yesterday so by rebasing you would have reliable CI feedback.

Rebased on master - CI is green again. :-)

@LysandreJik I refactored the tokenizer integration test of albert;

https://github.com/German-NLP-Group/transformers/blob/1894bcc5d116d0107150f4659551e6e21111d736/tests/test_tokenization_albert.py#L127

By adding a util class to `TokenizerTesterMixin`
https://github.com/German-NLP-Group/transformers/blob/1894bcc5d116d0107150f4659551e6e21111d736/tests/test_tokenization_common.py#L186

And also added an integration test to Barthez:
https://github.com/German-NLP-Group/transformers/blob/1894bcc5d116d0107150f4659551e6e21111d736/tests/test_tokenization_barthez.py#L99

What do you think about this "pattern"? Should I continue in that direction and add the other tokenizers?

I like it, I find it very clean!

@LysandreJik the reformer tokenizer integration test somehow fails:

```text
Expected :Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between Jax, PyTorch and TensorFlow.
Actual   :Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert provides general-purpose architectures (BERT, GPT-, RoBERTa, LM, DistilBert, LNet... for Natural Language nderstanding (NL and Natural Language Generation (NLG with over  pretrained models in  languages and deep interoperability between ax, PyTorch and TensorFlow.
```

Characters like ")" are missing from the vocab. They are converted to `0` or `<unk>`.
@LysandreJik I just pass in an simpler test text to make the test succeed.
Or should we investigate this stange error to discover a possible hidden bug?

@LysandreJik while we are here - can we remove this or is it some kind of "open todo"?

https://github.com/huggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/tests/test_tokenization_common.py#L178-L184

@LysandreJik @sgugger as discussed above the suggested integration tests are added to the sentencepiece tokenizers.

CI is green, IMO this is done and ready for merge.

Please have a look at the strange behavior of the reformer tokenizer: https://github.com/huggingface/transformers/pull/11737#issuecomment-850769064

And this question: https://github.com/huggingface/transformers/pull/11737#issuecomment-850776366


Pinging @patrickvonplaten regarding the Reformer test.

Regarding https://github.com/huggingface/transformers/pull/11737#issuecomment-850776366 we can just remove this

> Regarding #11737 (comment) we can just remove this

Done.

Ok - so this should be ready to be merged so I can continue with #11716 ?

Thanks again for all your work!