3.0.1: "unexpected keyword argument 'is_pretokenized'" when using batch_encode_plus() w/ Fast Tokenizers
Author: minimaxir
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

See title. Does not occur with "slow" tokenizers.

## To reproduce

```python
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
texts = ['I am a teapot', 'Short and stout']
tokenizer.batch_encode_plus(texts)
```

## Trace

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-a99759a2bc25> in <module>
      1 texts = ['I am a teapot', 'Short and stout']
      2 
----> 3 tokenizer.batch_encode_plus(texts)

/usr/local/lib/python3.8/site-packages/transformers-3.0.1-py3.8.egg/transformers/tokenization_utils_base.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding, truncation, max_length, stride, is_pretokenized, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   1813         )
   1814 
-> 1815         return self._batch_encode_plus(
   1816             batch_text_or_text_pairs=batch_text_or_text_pairs,
   1817             add_special_tokens=add_special_tokens,

/usr/local/lib/python3.8/site-packages/transformers-3.0.1-py3.8.egg/transformers/tokenization_gpt2.py in _batch_encode_plus(self, *args, **kwargs)
    365         )
    366 
--> 367         return super()._batch_encode_plus(*args, **kwargs)
    368 
    369     def _encode_plus(self, *args, **kwargs) -> BatchEncoding:

/usr/local/lib/python3.8/site-packages/transformers-3.0.1-py3.8.egg/transformers/tokenization_utils_fast.py in _batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_pretokenized, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    340             encodings = [encodings]
    341         else:
--> 342             encodings = self._tokenizer.encode_batch(
    343                 batch_text_or_text_pairs, add_special_tokens=add_special_tokens, is_pretokenized=is_pretokenized
    344             )

TypeError: encode_batch() got an unexpected keyword argument 'is_pretokenized'
```

## Environment info

- `transformers` version: 3.0.1
- `tokenizers` version: 0.8.0rc4
- Platform: macOS
- Python version: 3.8.2


Comments:
I must admit that I am confused here. I don't see how this is possible with one of the `0.8.0` version of `tokenizers`, they should have `is_pretokenized` as expected argument of both `encode` and `encode_batch`. Unfortunately, I was not able to reproduce it using the snippet you provided. 

Would it be possible that you share a Google Colab showing this bug?

Hmm, it doesn't appear to be happening on Colab: https://colab.research.google.com/drive/1TCGbkP63BAwKHFob9YvP_PE7-8cyAZ3b?usp=sharing

I have seen other weird issues with tokenizers so it might be an issue on my local config; however the versions are definitely correct, which is baffling.

If you can reproduce this on your local config, can you check `tokenizers.__version__`? I suspect that for some reason you had a previous version that didn't include this argument. Sometimes I have weird bugs with `pip` that I don't really understand either, and so it might be reporting a version different from the one actually loaded.

Ugh, yes, `tokenizers.__version__` returned 0.7.0. (seems like it didn't get overwritten when upgrading to `transformers` 3.0.1)

After uninstalling tokenizers (twice for some reason) via `pip`, then reinstalling 0.8.0rc4, it works now.

Thanks for the help! Closing.