new tokenizer backend breaks old code
Author: flozi00
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): any model

Language I am using the model on (English, Chinese ...): english

The problem arises when using:
* [x] the official example scripts: (give details below) QA script from official repo

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) Squad 1 and 2

## To reproduce

Steps to reproduce the behavior:

Just try to run the example script for QA task

The error output is too large to copy in here.
The new tokenizer backend is not backward compatible. The truncation is set to false by default. 
With the new tokenizer backend the truncation need to get initialized with True value, but in example or old codes there is often no possibility to set it while calling.
I think the feature conversion code is not updated to the new backend, right ?

## Expected behavior
It should train the model correctly

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: any
- Python version: 3.7-3.8
- PyTorch version (GPU?): 
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
Can you share the exact command line you ran and the last 20-30 lines of the error message?

The command given in the readme 


Truncation was not explicitely activated but `max_length` is provided a specific value, please use `truncation=True` to explicitely truncate examples to max length. Defaulting to 'only_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you may want to check this is the right behavior.

This is getting printed some hundreth of times


Truncation was not explicitely activated but `max_length` is provided a specific value, please use `truncation=True` to explicitely truncate examples to max length. Defaulting to 'only_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you may want to check this is the right behavior.

"""
Traceback (most recent call last):
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/site-packages/transformers/data/processors/squad.py", line 134, in squad_convert_example_to_features
    encoded_dict = tokenizer.encode_plus(
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1504, in encode_plus
    return self._encode_plus(
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 358, in _encode_plus
    return self._prepare_for_model(
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 573, in _prepare_for_model
    ids, pair_ids, overflowing_tokens = self.truncate_sequences(
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 675, in truncate_sequences
    assert len(ids) > num_tokens_to_remove
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "runsquad.py", line 827, in <module>
    main()
  File "runsquad.py", line 765, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
  File "runsquad.py", line 451, in load_and_cache_examples
    features, dataset = squad_convert_examples_to_features(
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/site-packages/transformers/data/processors/squad.py", line 326, in squad_convert_examples_to_features
    features = list(
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/multiprocessing/pool.py", line 420, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/home/a-ware/anaconda3/envs/transformers/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
AssertionError


Edit: http://prntscr.com/t3ak3t
The error occures while processing the datas

Same behaviour, any idea to debug it? 

As temporary fix I took an older fork without the new tokenizers logic and copied the new models to it

> Can you share the exact command line you ran and the last 20-30 lines of the error message?

It seems it it related with the issue I created here: https://github.com/huggingface/tokenizers/issues/307#issuecomment-647603101
cc: @sshleifer 

Yes, it the same error

> > Can you share the exact command line you ran and the last 20-30 lines of the error message?
> 
> It seems it it related with the issue I created here: [huggingface/tokenizers#307 (comment)](https://github.com/huggingface/tokenizers/issues/307#issuecomment-647603101)
> cc: @sshleifer



Still facing the same issue, did the fix work?

Was this issue solved? I am facing the same problem, it was working just fine this evening, and I re-installed transformers module on a new notebook, thats when it fetched the newer version and is causing this error, I've tested it with  Bert and gpt2, and it still presists. 
I read the code for the  tokenization_utils_base, here is where it's originating, based on the "if conditional block", i coded padding to be True, but still it gives same error.

```
def _get_padding_truncation_strategies(
        self, padding=False, truncation=False, max_length=None, pad_to_multiple_of=None, verbose=True, **kwargs
    ):
        """ Find the correct padding/truncation strategy with backward compatibility
            for old arguments (truncation_strategy and pad_to_max_length) and behaviors.
        """
        old_truncation_strategy = kwargs.pop("truncation_strategy", "do_not_truncate")
        old_pad_to_max_length = kwargs.pop("pad_to_max_length", False)

        # Backward compatibility for previous behavior, maybe we should deprecate it:
        # If you only set max_length, it activates truncation for max_length
        if max_length is not None and padding is False and truncation is False:
            if verbose:
                logger.warning(
                    "Truncation was not explicitely activated but `max_length` is provided a specific value, "
                    "please use `truncation=True` to explicitely truncate examples to max length. "
                    "Defaulting to 'only_first' truncation strategy. "
                    "If you encode pairs of sequences (GLUE-style) with the tokenizer you may want to check this is the right behavior."
                )
            truncation = "only_first"
```

Are you installing it from source?

@mrm8488 No, currently not, its from pip, could it be that the pip isn't getting the updated code, the version shows transformers-3.0.0, and I remember yesterday it wasn't 3.0.0

Closing as adressing @llStringll's issue in #5377.