[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies
Author: thomwolf
Pull request: yes

# What does this PR do?

Both the [SentencePiece](https://github.com/google/sentencepiece) and [Tokenizers](https://github.com/huggingface/tokenizers) libraries can limit the users:
- `sentencepiece` is not available on Conda on every plateform and one of the reason `transformers` is not on Conda
- `tokenizers` cannot be used inside some labs which need to build all from source and don't have a Rust tooling.

This PR aim at making both optional leveraging the addition of SentencePiece algorithms in Tokenizer.

Note: at least one of `sentencepiece` and `tokenizers` will be required to use the SentencePiece tokenizers. `tokenizers` is also required to use the Fast tokenizers.

Main changes in the library organization:
- fast tokenizers are now separated in `tokenization_XXX_fast.py` files
- a `convert_slow_tokenizer.py` file host conversion methods for a slow to a fast tokenizer but a direct path from a `tokenizers` serialization file is favored when such a file is available.
- the test suite for slow and fast tokenizers are now gathered in a single test suite.

Main new requirements for the tokenizers to pass the new test suite:
- at least one default vocabulary checkpoint (and max length) should be provided, it is used for the deep tests
- the fast tokenizer should have an explicit `tokenizer_file` keyword argument with a default to `None` (we check that to be sure all the fast tokenizer can accept the new serialization format.

To-add:
- when the documentation for `tokenizers` is ready: add a lot of link on how to build and add a fast tokenizer
- add a detailed explanation on how to add a fast tokenizer in the library

This PR also:
- add a `__repr__` for the tokenizers (finally...)
- add a `name_or_path` attribute to the models and tokenizers giving the shortcut name or the path of the pretrained checkpoint used for instantiation
- update the fast tokenizer to use (when possible) the new serialization format of the `tokenizers` library, falling back on the old diverse set of saving format if not available.
- clean up the tests for the fast tokenizers to bring them in the common tokenizer tests

Fixes #7402 #5100 (and maybe others)

## Before submitting
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? 


Comments:
Ok ready for review @LysandreJik @sgugger.

It's pretty big sorry.

For now sentencepiece is still in the requirements as removing it has some effect on the pipeline tests and I think it's probably good to study this in a separate future PR.

There is no breaking change apart from the fact that importing the `**Fast` tokenizer directly from the `transformers.tokenization_xxx` is not possible anymore, they should be imported from `transformers` (the best and most robust choice) or from their new respective location at `transformers.tokenization_xxx_fast`.

Ok the `examples/seq2seq/test_seq2seq_examples.py::test_finetune[stas/tiny-wmt19-en-de]` is working now.
I'll address the other comments and we can merge on Monday.

@thomwolf, could you please assign defaults that are different from "stas/tiny-wmt19-en-de" entry and its contents? Otherwise it defeats the purpose of testing with this model, since defaults are used instead.

Alternatively, I will need to create a new tiny model with different config and change tests to use that instead.

Once this is done let's add this test that I tried to add here: https://github.com/huggingface/transformers/pull/7860 - I expanded it below a bit to do better testing:

```
diff --git a/tests/test_tokenization_fsmt.py b/tests/test_tokenization_fsmt.py
index c3e08d56..833b1742 100644
--- a/tests/test_tokenization_fsmt.py
+++ b/tests/test_tokenization_fsmt.py
@@ -24,6 +24,7 @@ from transformers.tokenization_fsmt import VOCAB_FILES_NAMES, FSMTTokenizer

 from .test_tokenization_common import TokenizerTesterMixin

+FSMT_TINY = "stas/tiny-wmt19-en-de"

 class FSMTTokenizationTest(TokenizerTesterMixin, unittest.TestCase):
     tokenizer_class = FSMTTokenizer
@@ -86,6 +87,13 @@ class FSMTTokenizationTest(TokenizerTesterMixin, unittest.TestCase):
     def tokenizer_en_ru(self):
         return FSMTTokenizer.from_pretrained("facebook/wmt19-en-ru")

+    def test_online_tokenizer_config(self):
+        """this just tests that the online tokenizer files get correctly fetched and
+        loaded via its tokenizer_config.json and it's not slow so it's run by normal CI
+        """
+        tokenizer = FSMTTokenizer.from_pretrained(FSMT_TINY)
+        self.assertListEqual([tokenizer.src_lang, tokenizer.tgt_lang], ["en", "de"])
+
     def test_full_tokenizer(self):
         """ Adapted from Sennrich et al. 2015 and https://github.com/rsennrich/subword-nmt """
         tokenizer = FSMTTokenizer(self.langs, self.src_vocab_file, self.tgt_vocab_file, self.merges_file)

```

Thanks.


Yes feel free to create another model for fsmt @stas00.

Ok this big PR is ready for merge as soon as possible (with regards to other PR merges not absolute time) so it doesn't drift too much.

There are some `~transformers.tokenization_utils_base.PreTrainedTokenizer` left (and same with fast) but that's an easy pattern to search for a subsequent PR.

Ok then I'm merging this PR and continuing in another one to:
- add CI tests for the package without sentencepiece and tokenizer
- remove sentencepiece as a required dependency
- switch to fast tokenizers by default
- fix the remaining doc patterns that you mentioned

On the topic of `from_pretrained` logic, we could (should probably be another PR):
- add a test that the config of the tokenizers is used as mentioned by @stas00
- we could probably remove the hard-coded configs at the same time
- switch to the cloud-front links like the models for faster dowloads

**edited**: thanks to @sshleifer  - I needed to `pip install -e ".[dev]"` to update the new dependencies. that fixed the issues.

----------

I'm getting a massive amount of identical failures after this merge, primarily:

```
_____________________________________________ XLNetTokenizationTest.test_num_special_tokens_to_add_equal _____________________________________________
[gw1] linux -- Python 3.8.5 /home/stas/anaconda3/envs/main-38/bin/python

self = <tests.test_tokenization_xlnet.XLNetTokenizationTest testMethod=test_num_special_tokens_to_add_equal>

    def test_num_special_tokens_to_add_equal(self):
        for tokenizer, pretrained_name, kwargs in self.tokenizers_list:
            with self.subTest("{} ({})".format(tokenizer.__class__.__name__, pretrained_name)):
>               tokenizer_r = self.rust_tokenizer_class.from_pretrained(pretrained_name, **kwargs)

tests/test_tokenization_common.py:1896: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/tokenization_utils_base.py:1588: in from_pretrained
    return cls._from_pretrained(
src/transformers/tokenization_utils_base.py:1661: in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
src/transformers/tokenization_xlnet_fast.py:142: in __init__
    super().__init__(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[AttributeError("'XLNetTokenizerFast' object has no attribute 'name_or_path'") raised in repr()] XLNetTokenizerFast object at 0x7f2ccdaaee80>
args = (), kwargs = {'additional_special_tokens': ['<eop>', '<eod>'], 'bos_token': '
', 'cls_token': '<cls>', 'do_lower_case': False, ...}
slow_tokenizer = None
fast_tokenizer_file = '/home/stas/.cache/torch/transformers/d152c146766f0a31888c4c9c0dcf82e42e42d09bf818bb74e126f2420cbd36c4.ecf1d38c0b94010f431264b9ded85217342f84c7bdae79b0472f7cd20b94052d'

    def __init__(self, *args, **kwargs):
        slow_tokenizer = kwargs.pop("__slow_tokenizer", None)
        fast_tokenizer_file = kwargs.pop("tokenizer_file", None)
    
        if fast_tokenizer_file is not None:
            # We have a serialization from tokenizers which let us directly build the backend
>           fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
E           Exception: data did not match any variant of untagged enum PyNormalizerTypeWrapper at line 1 column 318041
```
do I need to remove cache or something? I won't test this until you tell me to in case you need someone with the old cache to test that it can recover from this.

A total of 90 failed tests with this error.

You should update `tokenizers` to the main PyPi version @stas00 
```
pip install tokenizers --update
```