AutoTokenizer from pretrained BERT throws TypeError when encoding certain input
Author: sorenmulli
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  4.3.2
- Platform: Arch Linux
- Python version: 3.9.1
- PyTorch version (GPU?): 1.7.1, no
- Tensorflow version (GPU?): Not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
Guess from git blame: @LysandreJik , @thomwolf @n1t0 

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
When I use a pretrained BERT tokenizer, it throws a TypeError on singleton input or input containing ø/æ/å.

It was discovered when I used the pretrained `Maltehb/danish-bert-botxo` which would fail in the below way on any input containing Danish characters (ø/æ/å), but I also realized that it happens with the standard `bert-base-uncased` as shown below.

Steps to reproduce the behavior:
1. Run these line
```py
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokenizer.encode(["hello", "world"])                          # <--- This works
tokenizer.encode(["hello"])                                   # <--- This throws the below shown stack trace
tokenizer.encode(["dette", "er", "en", "sø"])                 # <--- This throws the same error
```
Stack trace
```py
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-ef056deb5f59> in <module>
----> 1 tokenizer.encode(["hello"])

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_base.py in encode(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, return_tensors, **kwargs)
   2102                 ``convert_tokens_to_ids`` method).
   2103         """
-> 2104         encoded_inputs = self.encode_plus(
   2105             text,
   2106             text_pair=text_pair,

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_base.py in encode_plus(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2418         )
   2419 
-> 2420         return self._encode_plus(
   2421             text=text,
   2422             text_pair=text_pair,

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py in _encode_plus(self, text, text_pair, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    453 
    454         batched_input = [(text, text_pair)] if text_pair else [text]
--> 455         batched_output = self._batch_encode_plus(
    456             batched_input,
    457             is_split_into_words=is_split_into_words,

~/.venv/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py in _batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose)
    380         )
    381 
--> 382         encodings = self._tokenizer.encode_batch(
    383             batch_text_or_text_pairs,
    384             add_special_tokens=add_special_tokens,

TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I expect the model not to throw a type error when the types are the same.
I also expected that the tokenization would produce id's.

[This issue](https://github.com/alexandrainst/danlp/issues/113) is caused by the above 
I am grateful for the software and thank you in advance for the help!


Comments:
Hello! Thank you for opening an issue with a reproducible example, it helps a lot.

The issue here is that you're using the `encode` method to encode a batch, which it can't do. Encode only encodes single sequences, and can accept a "batch" of two because it processes them as two independent sequences that should be joined together, for example for text-classification where you would want to classify the relationship between two sequences (tasks like Next Sentence Prediction from BERT or Sentence Ordering Prediction ALBERT).

The method you're looking for is the `__call__` method of the tokenizer, which handles exactly all the use-cases you've mentioned, and is the recommended API for tokenizers:

```py
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokenizer(["hello", "world"])                          # <--- This works
tokenizer(["hello"])                                   # <--- This works too :)
tokenizer(["dette", "er", "en", "sø"])                 # <--- This works as well!
```

[Here is the documentation](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.__call__) for that method, hope that helps!

Thank you very much for this good explanation which clearly resolves my problem.

Do you by any chance know whether this behaviour changed in the last years time?
The transformers-based repos [NERDA](https://github.com/ebanalyse/NERDA) and [danlp](https://github.com/alexandrainst/danlp) seem to rely on `tokenizer.encode` to be working as you show the call method does, and as such fail on the current version, but work on 3.5.1 (https://github.com/alexandrainst/danlp/issues/113)

I believe the `encode` method never accepted batches as inputs. We introduced `encode_plus` and `batch_encode_plus` down the road, the latter being the first to handle batching.

While these two methods are deprecated, they're still tested and working, and they're used under the hood when calling `__call__`.

What is happening here is that v3.5.1 is treating your input as individual words (but by all means it shouldn't as the `is_split_into_words` argument is `False` by default), rather than as different batches, I was mistaken in my first analysis. Something did change between version v3.5.1 and v4.0.0, all the breaking changes are documented in the [migration guide](https://huggingface.co/transformers/migration.html).

If you want to get back to the previous behavior, you have two ways of handling it:

- Specify that you don't want a fast tokenizer. The main change affecting you here is that the `AutoTokenizer` returns a fast tokenizer by default (in Rust) rather than the python-based tokenizer. You can change that behavior with the following:
```py
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=False)
```
- The behavior you're relying on here is the `is_split_into_words` parameter: you're passing it a list of words, rather than a sequence of words. That it worked in previous versions seems like a bug to me, here's how you would handle it now (works with a fast tokenizer):
```py
tokenizer(["hello", "world"], is_split_into_words=True)
tokenizer(["hello"], is_split_into_words=True)
tokenizer(["dette", "er", "en", "sø"], is_split_into_words=True)
```

Thank you, the `is_split_into_words` clears up the confusion between batches and tokens clearly for me!

Hi @LysandreJik 

I am excpericing the same error:
`'TypeError: TextEncodeInput must be Union[TextInputSequence,Tuple[InputSequence, InputSequence]]'`

while running below code:
```python
self.tokenizer.encode_plus(example[0],
                          add_special_tokens=True,
                          padding='max_length',
                          max_length=max_length,
                          return_attention_mask=True,
                          return_tensors='pt')
```

`example[0]` is list of Int which I encoded:
[49518, 111, 22560, 20, 1112, 128, 29, 568, 7, 7244, 10, 10905, 111, 12396, 3781, 111, 4878, 1087, 396, 10, 812, 111, 3077, 629, 847, 202, 3607, 490, 5, 3302, 9, 17890, 154, 10, 3077, 629, 4878, 42, 76, 479, 10130, 273, 363, 2156, 5, 1112, 2763, 8176, 111, 262, 7, 7244, 41, 2319, 68, 508, 4, 245, 325, 2450, 14, 56, 57, 12850, 9, 2213, 9, 7668, 14, 74, 33, 23398, 2156, 1195, 87, 20546, 2156, 5, 752, 1229, 3781, 479, 2589, 6040, 17811, 28455, 5, 1087, 7, 18720, 3633, 14, 24, 21, 33602, 19, 780, 111, 773, 629...

Now i want to pad it and get the attention back.
In the docs it mentioned that i can send List[Int]
what I am missing ?


Hi @shon-otmazgin could you open a new issue with a reproducible code example + full stack trace so that we can take a look? Thanks!

Taking a look at it, I believe the documentation is wrong here and the fast tokenizers handle strings as inputs. Have you tried using `prepare_for_model` for your use-case?

I will take a look on `prepare_for_model ` this is new to me.
`prepare_for_model ` accept list of input_ids, can pad and return attention mask? 

So we dived into it with @n1t0 and actually the problem here is slightly complex. The slow & fast tokenizers have roughly the same API with a few excptions, and this is one of them: the fast tokenizers are great at handling strings and at being extremely efficient with a bunch of features (offsets is one example of a really powerful feature), but they're not made to handle lists of ints.

In this particular case, while I think it is theoretically possible with fast tokenizers methods by using some private methods, it seems you would be way better off to use a slow tokenizer to achieve what you're looking for.

But this begs the question: is there a way you could share your use-case so that we could study it and understand why you need to pass already processed lists of ints to the tokenizer, instead of tokenizing the text and relying on the information within the encoding?

Here the fast tokenizers would probably be way more efficient at handling this use-case in a one step process, rather than the two step process we're trying to achieve here.

Also, I'm seeing the following in the docs for `encode_plus`:
![image](https://user-images.githubusercontent.com/30755778/113611903-25f58f00-961d-11eb-89f6-dbc5102f7dca.png)

and for `batch_encode_plus`:
![image](https://user-images.githubusercontent.com/30755778/113611980-3c9be600-961d-11eb-883c-bfc30dbb3dc5.png)

Is there a docstring we've forgotten somewhere that tells this is also supported for fast tokenizers?



I tell you what happened:
I worked on version 3.3.1 which by default `use_fast=False` for `AutoTokenizer`.
I upgraded to version 4.4.2 and that break. -> `use_fast `changed to `True `for `AutoTokenizer`.

@LysandreJik thank you very much for your help. appreciate that :)