[breaking|pipelines|tokenizers] Adding slow-fast tokenizers equivalence tests pipelines - Removing sentencepiece as a required dependency
Author: thomwolf
Pull request: yes

# What does this PR do?

**Breaking**: Auto-tokenizers and pipelines:
- switch to `use_fast=True` by default (Fast tokenizers by default)
=> The main expected breaking change is **the handling of overflowing tokens** which is different between slow and fast tokenizers.
- removing sentencepiece from the required dependencies (in some special case this may require you to install `sentencepiece` in addition to the normal install).

Pipelines:
- Add slow/fast tokenizers equivalence tests in pipelines
- upgrade QA/NER processing pipeline to handle fast tokenizers
- remove `test_pipelines_dialog.py` which was a duplicated test file

Tokenizers:
- Update and add new alignement method in `BatchEncoding` 

Dependencies:
- upgrade to tokenizers==0.9.4 to allow QA processing with fast tokenizers
- remove sentencepiece from the required dependencies 

Misc:
- Fix bug in RobertaFast and test for XLM-Prophetnet and RAG

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
On hold for now until we update the input/output of the alignement methods in `tokenizers` to handle pairs of input sentences following internal discussion with @n1t0 and @Narsil.

Remaining error on the CI (NER pipeline not working for slow tokenizers) should be solved by #8364  

Edit: ok solved now that #8364 is merged

examples/seq2seq/finetune.py is failing after this PR:

```
CUDA_VISIBLE_DEVICES=0 pytest -sv examples/seq2seq/test_seq2seq_examples.py::TestTheRest::test_finetune_0_patrickvonplaten_t5_tiny_random
```


```
Traceback (most recent call last):
  File "finetune.py", line 442, in <module>
    main(args)
  File "finetune.py", line 409, in main
    trainer: pl.Trainer = generic_train(
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/lightning_base.py", line 398, in generic_train
    trainer.fit(model)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 444, in fit
    results = self.accelerator_backend.train()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 63, in train
    results = self.train_or_test()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in train_or_test
    results = self.trainer.train()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 493, in train
    self.train_loop.run_training_epoch()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 554, in run_training_epoch
    for batch_idx, (batch, is_last_batch) in train_dataloader:
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 80, in profile_iterable
    value = next(iterator)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 46, in _with_is_last
    last = next(it)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 519, in __next__
    data = self._next_data()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1169, in _next_data
    return self._process_data(data)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1195, in _process_data
    data.reraise()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/utils.py", line 251, in collate_fn
    batch_encoding: Dict[str, torch.Tensor] = self.tokenizer.prepare_seq2seq_batch(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/bart/tokenization_bart_fast.py", line 127, in prepare_seq2seq_batch
    model_inputs: BatchEncoding = self(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 2319, in __call__
    return self.batch_encode_plus(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 2504, in batch_encode_plus
    return self._batch_encode_plus(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/gpt2/tokenization_gpt2_fast.py", line 167, in _batch_encode_plus
    return super()._batch_encode_plus(*args, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_fast.py", line 433, in _batch_encode_plus
    return BatchEncoding(sanitized_tokens, sanitized_encodings, tensor_type=return_tensors)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 242, in __init__
    n_sequences = encoding[0].n_sequences
AttributeError: 'tokenizers.Encoding' object has no attribute 'n_sequences'

Exception ignored in: <function tqdm.__del__ at 0x7f60b55d7b80>
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1128, in __del__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1341, in close
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1520, in display
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1131, in __repr__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1481, in format_dict
TypeError: cannot unpack non-iterable NoneType object
```

Hi @stas00, it seems you don't have the latest tokenizer version!

Bummer! Thank you for identifying that, @LysandreJik - I confirm updating `tokenizers` fixes it.

Since this happened more than once now, would it be possible to add and maintain run-time checks as we have done here:

https://github.com/huggingface/transformers/blob/dd52804f5fce0a568ffbb3dc7fd088d2de0a0e56/examples/lightning_base.py#L38-L47

How a developer is to know that they need to update a dependency of a project otherwise? `git pull` doesn't trigger `pip install -e ".[dev]"`

Except the code above should emit an error on failed check. It should be an error in `lightning_base.py` too, but I couldn't convince @sshleifer to make it so. Warnings are plentiful and this approach of using a warning just doesn't serve its purpose, IMHO.


I like the function (though we can expand its name and replace `ver` by `version` inside to make it more readable ;-) ) and I think we could have a dynamic check at init. I don't see any problem with throwing an error if the minimum version of tokenizers isn't installed, but maybe @LysandreJik or @patrickvonplaten have a different opinion.

Oh, any name works, it was just code first, then I thought that it'll eventually end up being a reusable function, so the half-baked function emerged.

I'm glad you agree that it should assert if the minimal requirements aren't met.

Technically we should then add `packaging` and `pkg_resources` to the project dependencies, but since they are prerequisites for `setuptools` - every user should already have them.

And we will need 2 versions of errors: 
* `Try: pip install -r examples/requirements.txt` for examples, 
* `Try: pip install transformers -U"` for the core.

Agree! I actually also already ran into the same issue @stas00 -> I'm definitely in favor of such a function! Should we run the function at `import transformers` already ? 

So did I, and I thought everything was broken until @sgugger showed me the way. Your function would definitely help in that regard @stas00 :)

> Should we run the function at `import transformers` already ?

Two ways I can see:

1. right where the corresponding import is done - so it's easier to see the specific requirement - but it could be scattered - but it could be more difficult to maintain. Ideally, python would have `import module 1.2.3`, like some other languages have, but it doesn't at the moment.
2. all in one place, at the very top of `__init__.py`, all requirements next to each other so it's easy to maintain. This check relies on packaging tools - i.e. derives it from the `site-packages` dir, so it shouldn't even load the module. i.e. we don't try to look for `xxx.__version__` here, since not all packages have it.

I'd say (2) is the easier way.

Last night I was dreaming of a trigger feature, where if git sees `setup.py` modified it'd alert someone to update `__init__.py` requirements - but it was a dream.


I think it can be in any file imported by the `__init__` (as long as the function is executed), so we could also have this in `file_utils.py`. Though the `__init__` is fine by me too if you think it's better.

Sure, then let's add a dedicated module then? It'd be the most simple/intuitive then

```
$ cat version_requirements.py
require("tokenizers", "1.2.3")
...
```
and `__init__.py`:
```
import .version_requirements
```

hear, hear! if we have such a file then we can even use it to feed `setup.py`! so we have a single place where we edit all the minimal version requirements and don't need to touch `setup.py` and potentially forget to sync requirements.

In which case I'd use a dict and feed it to `require_version` (or whatever we end up calling it).

Clearly setup has a lot of optional things, so perhaps then we load this file at the end of __init__ and only check versions for the things that got loaded?

or we just test only the package names that we know we need to check, but use that dict for setup.py's needs.

Let me know if these ideas are an overkill.

Here is a quick prototype to what I'm thinking:
```
$ cat src/transformers/version_requirements.py
min_vers = dict(
    tokenizers: "==0.9.4",
    tqdm: ">=4.27",
    jaxlib: "==0.1.55",
)
run_time_keys = "tokenizers tqdm".split()
for k in run_time_keys:
    require_min_ver(k, min_vers[k])

$ cat setup.py
from version_requirements import min_vers
# of course we won't hardcode each entry - this is a just to demonstrate
extras["flax"] = [f"jaxlib{min_vers{'jax_lib']}", ...

```

so you can see the dictionary has all the versions, but we actively check only the versions that are non-optional.


One downside of this is that it would move dependencies out of the setup.py (which is where people would expect to see them). Do you think there is a way to structure this so the one place we look at minimum version is the setup? It would be less surprising I think. 

I agree.

We could have all the version requirements defined in `setup.py` and when it's run it'd update `src/transformers/version_requirements.py` instead. Then we would actually want 2 files under transformers - one that  `setup.py` will maintain - it will be just a dict dump - so that it could overwrite the file completely and another for the selective run-time checks that would refer to the first file generated by setup, since we will only check a handful of these many dependencies at run time.

```
$ cat setup.py
min_vers = dict(
    tokenizers: "==0.9.4",
    tqdm: ">=4.27",
    jaxlib: "==0.1.55",
)

# add code to dump min_vers dict into `src/transformers/version_requirements.py`

# of course we won't hardcode each entry - this is a just to demonstrate
extras["flax"] = [f"jaxlib{min_vers{'jax_lib']}", ...


$ cat src/transformers/version_requirements.py
# AUTOGENERATED - MODIFY setup.py INSTEAD! #
min_vers = dict(
    tokenizers: "==0.9.4",
    tqdm: ">=4.27",
    jaxlib: "==0.1.55",
)

$ cat src/transformers/version_run_time_check.py
from .version_requirements import min_vers

# define which module versions we always want to check (only a few)
run_time_keys = "tokenizers tqdm".split()
for k in run_time_keys:
    require_min_ver(k, min_vers[k])

$ cat src/transformers/__init__.py
import .version_run_time_check
```

This is of course all just a visual prototype.


If you want to tackle this, please go ahead with something along these lines. We can refine more on an actual PR.

OK, I will make a partial sub-set of modules and when you like how it looks expand it to all modules.

Is there any way to workaround the version check?
I want to use some features from `tokenizers 0.10`, but Transformers raise `VersionConflict`.

Surely, this can cause some very non-obvious bugs, but at least I'll be able to work with my code before the new version of Transformers is released.

I think this is a very reasonable need, @Guitaricet. But it's probably best to discuss it in a dedicated issue. Could you please file a [feature request](https://github.com/huggingface/transformers/issues/new/choose) and let's see what others would think?

I'd say an env var to override the checks should do the trick. Should be easy to add if the others agree with having it.