Request: TokenClassification pipeline batch processing over a sequence of already tokenised tests
Author: davidsbatista
Pull request: no

I have a fine-tuned `model` which performs token classification, and a `tokenizer` which was built as:

`tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")`

and this works fine in a pipeline when processing a single document/message:

```
nlp = pipeline(
    "token-classification",
    model=model,
    tokenizer=tokenizer,
    aggregation_strategy="first",  # max, none, simple, average
    binary_output=True,
    ignore_labels=[],
)

text = ["Hello", "this", "is", "a", "single", "tokenized", "message"]

for token in nlp(text):
	print(token)

[{'entity_group': 'LABEL_2', 'score': 0.07955505, 'word': 'Hello', 'start': 0, 'end': 5}]
[{'entity_group': 'LABEL_2', 'score': 0.06315145, 'word': 'this', 'start': 0, 'end': 4}]
[{'entity_group': 'LABEL_2', 'score': 0.08200004, 'word': 'is', 'start': 0, 'end': 2}]
[{'entity_group': 'LABEL_2', 'score': 0.07786057, 'word': 'a', 'start': 0, 'end': 1}]
[{'entity_group': 'LABEL_3', 'score': 0.056751117, 'word': 'single', 'start': 0, 'end': 6}]
[{'entity_group': 'LABEL_3', 'score': 0.10323574, 'word': 'tokenized', 'start': 0, 'end': 9}]
[{'entity_group': 'LABEL_3', 'score': 0.09412522, 'word': 'message', 'start': 0, 'end': 7}]
```

If I know try to pass a sequence of messages:

````
text = [
    ["Hello", "this", "is", "a", "single", "tokenized", "message"], 
    ["another", "tokenized", "message"], 
    ["short", "message"]
]
````
I was expecting I could do something like this:

```
for msg in nlp(text):
    for entity in nlp(msg):
	print(entity)
```

but I always end up with a `ValueError`:

`ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.`

Even if I initialise the tokenizer to be passed to the pipeline as this, I always get the same `Value Error`

```
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased", model_max_length=512, is_split_into_words=True, padding=True, truncation=True)

nlp = pipeline(
    "token-classification",
    model=model,
    tokenizer=tokenizer,
    aggregation_strategy="first"
    binary_output=True,
    ignore_labels=[],
)
```

Also if I "pad" the tokenized input like this

```
text = [
   ["Hello", "this", "is", "a", "single", "tokenized", "message"],
   ["another", "tokenized", "message", "[PAD]", "[PAD]", "[PAD]", "[PAD]"],
   ["short", "message", "[PAD]", "[PAD]", "[PAD]", "[PAD]", "[PAD]"]
]

for x in nlp(text, batch_size=3):
    print(x)
```

From a quick read of the code of the pipeline it seems there's no way to instruct the tokeniser inside the TokenClassificationPipeline that the input is needs to be padded/truncated AND is already tokenised.


Comments:
Hi @davidsbatista ,

Sorry but you cannot pass pretokenized text to the pipelines. The error message is wrong.

You need to pass direct sentences to the pipeline.

```python
sentences = [
   "Hello this is a single tokenized message",
  "another tokenized message"
  "short message"
]
for tokenized in nlp(sentences):
    print(tokenized)
```

Because pipeline manages the tokenizer itself, it will chunk the sentence as the model expects, and there's no way to force other boundaries by "pretokenizing" sentences. 

Why do you want to pass pretokenized sentences ? Was something wrong when attempting to send real sentences ?

Cheers !

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.