Repository: huggingface/transformers

Issue:
Potential bug: Tokens with punctuation are re-tokenized although I've set `is_split_into_words=True`
## Environment info
- `transformers` version: 4.5.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
tokenizers: Hi @LysandreJik!

## Information
I am working on a token classification task where my input is in the following format: 

```
texts = [['Foo', 'bar', '.'], ['Hello', 'world', '.']]
tags = [['B-ENT, 'I-ENT', 'O'], ['O', 'O, 'O']]
```

- **Model:** `bert-large-cased`
- **Tokenizer:** `BertTokenizerFast`. I align tokens with their tags as in this [tutorial](https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities).

## Problem
Although I've set `is_split_into_words=True` in the tokenizer, tokens containing punctuation are tokenized.

## To reproduce
I reproduced the issue in this [Google Colab notebook](https://colab.research.google.com/drive/1mNJ-T6kOaC5_a8C3T_qCXBNFP5WYC5oi?usp=sharing).

## Expected behavior
Since I've set `is_split_into_words=True`, I would expect the tokenizer to keep the tokens as they are and split them into subwords with `##`. For example, if a token is `'foo(bar'`, I would expect it to stay that way, instead of being split into `['foo', '(', 'bar']`.

Thanks a lot for reading the issue! 


Discussion:
The `is_split_into_words` should be set to skip pre-tokenization (splitting on whitespace), not tokenization. This flag should be set to `True` if you have split your text into individual words, and you're now looking to have each word split into tokens and converted to IDs.

This seems to be unclear from the documentation, we'll work on improving this.
Makes sense, thank you for the clarification! I'd be happy to work on this if needed.
Yes, we would welcome such a contribution! I guess we would need to find all occurrences of that `is_split_into_words` parameter and clarify that pre-tokenization isn't tokenization as one could expect. 

We would gladly welcome a PR!
Great, I will work on this next week! Thanks again for the help!

Resolving patch:
diff --git a/src/transformers/models/tapas/tokenization_tapas.py b/src/transformers/models/tapas/tokenization_tapas.py
--- a/src/transformers/models/tapas/tokenization_tapas.py
+++ b/src/transformers/models/tapas/tokenization_tapas.py
@@ -172,8 +172,9 @@ def whitespace_tokenize(text):
                 length is required by one of the truncation/padding parameters. If the model has no specific maximum
                 input length (like XLNet) truncation/padding to a maximum length will be deactivated.
             is_split_into_words (:obj:`bool`, `optional`, defaults to :obj:`False`):
-                Whether or not the input is already pre-tokenized (e.g., split into words), in which case the tokenizer
-                will skip the pre-tokenization step. This is useful for NER or token classification.
+                Whether or not the input is already pre-tokenized (e.g., split into words). If set to :obj:`True`,
+                the tokenizer assumes the input is already split into words (for instance, by splitting it on
+                whitespace) which it will tokenize. This is useful for NER or token classification.
             pad_to_multiple_of (:obj:`int`, `optional`):
                 If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
                 the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -643,7 +643,9 @@ def prepare_for_tokenization(
             text (:obj:`str`):
                 The text to prepare.
             is_split_into_words (:obj:`bool`, `optional`, defaults to :obj:`False`):
-                Whether or not the text has been pretokenized.
+                Whether or not the input is already pre-tokenized (e.g., split into words). If set to :obj:`True`,
+                the tokenizer assumes the input is already split into words (for instance, by splitting it on
+                whitespace) which it will tokenize. This is useful for NER or token classification.
             kwargs:
                 Keyword arguments to use for the tokenization.
 
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -1286,8 +1286,9 @@ def all_special_ids(self) -> List[int]:
                 returned to provide some overlap between truncated and overflowing sequences. The value of this
                 argument defines the number of overlapping tokens.
             is_split_into_words (:obj:`bool`, `optional`, defaults to :obj:`False`):
-                Whether or not the input is already pre-tokenized (e.g., split into words), in which case the tokenizer
-                will skip the pre-tokenization step. This is useful for NER or token classification.
+                Whether or not the input is already pre-tokenized (e.g., split into words). If set to :obj:`True`,
+                the tokenizer assumes the input is already split into words (for instance, by splitting it on
+                whitespace) which it will tokenize. This is useful for NER or token classification.
             pad_to_multiple_of (:obj:`int`, `optional`):
                 If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
                 the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).


Test patch: