processing texts longer than 512 tokens with token-classification pipeline 
Author: oliverguhr
Pull request: no

# 🚀 Feature request

Currently, the token-classification pipeline truncates input texts longer than 512 tokens. 
It would be great if the pipeline could process texts of any length.

## Motivation

This issue is a result of this feature request:
huggingface/hub-docs#11

As suggested I am tagging @Narsil here. 

## Your contribution

For my [punctuation prediction model](https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large), I wrote an inference code for this task.
This code segments a longer text into 512 tokens chunks with some overlap and 
reconstructs the final output string. The code however is rather complex and tailored to 
my punctuation prediction use case. So I am not sure if this is useful here. 


Comments:
Hi @oliverguhr ,

So this specific pipelines truncates by default, and changing that would mean a breaking change.

The problem with "chunking" on this pipeline is that models usually need context before and after a token to get proper context to get a useful classification.
That means, than in order to achieve this, the pipeline should be able chunk into multiple parts of text, with overlap (usually called stride, at least in vision, but we borrowed that name in pipelines too). Run the `token-classification` on all the parts, and then resolve conflicts to give a final output.

Two things to have in mind:

- Conflict resolution is the tricky part and I don't think we can find a win always solution. It means that it's likely that the final result won't be as good as if we could have sent the full text to the model.
Consider the Example:

```
[The best city in the world is New York, because of Central Park]
--------------------------------------
                                ----------------------------------------
```

Both parts "see" New York, but one said it's a location while the other says it's nothing with greater confidence for instance. How would you resolve this ? What if New and York get different predictions for both parts. There's already a relatively complex resolution for tokens in the single case, this would make it more complex (that's ok).

- This can never be a default, always an opt-in. The pipelines' goal is to abstract away everything dealing with Tensors and logits, and manipulate simpler objects like strings and dicts. However, models being limited in range **is** a core limitation of those models (some models have an extremely long range) that we don't want to hide away (or at least not by default) especially if the result is not going to be the best one in all situations. Currently at least, we want users to know when there's a limitation and explicitly activate long range fallbacks. The only exception to the rule is `question-answering` currently.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I've been looking for a similar feature to this, and also written custom, non-pipeline code to handle it so far.
I can't afford to throw away text past the max length, so use the stride as you describe to include additional context. This would be really handy to have included in the pipeline, given that the functionality to wrap overflowing tokens is in the tokenizer.

I agree that conflict resolution is the difficult part here, and will add complexity. I'd see it as similar to the way entities are aggregated from tokens. The pipeline would allow strategies to be passed, where this may use strategies such as "greatest confidence", "prefer entities", "prefer first" etc. (all pretty naive strategies, I know). This would let the user decide how truncation is, or isn't applied, with a flag to discard anything past the model max length (or some overriding max length).

I've been trying to implement this in a TokenClassificationPipeline subclass recently. If this truncation/overflowing were to be included in the pipeline, where would it go? Would it be part of the `preprocess` function? Or does it also need to be included in the `collate_fn` of the base? The `overflow_to_sample_mapping` key currently breaks the existing `collate_fn` as its shape is different, so something needs to change there.

>  If this truncation/overflowing were to be included in the pipeline, where would it go? Would it be part of the preprocess function?

Hi @jammie19, any help would be super welcome here ! 

You would need to switch from `Pipeline` to `ChunkPipeline` (changes assumption of 1input-1 model pass, to 1 input-n model pass, while preserving batching, collation etc..).
Codewise this change is not too hard (going for `return` to `yield`, you can check `ZeroShotClassificationPipeline` for inspiration.
That should take care of the `collate_fn` for you (it's in `pipelines.pt_utils`) and `uncollate_fn` if you will too (since batching doesn't have to align with the number of items your input is creating). 

So preprocess would handle the chunking/overflowing+stride mecanisms, leveraging the tokenizer, but the post-process would be in charge of bringing the pieces together, `AutomaticSpeechRecognition` is another `ChunkPipeline` which has more postprocessing logic to stitch outputs.

Overall without actually digging into this work, it's hard to give you 100% confident advice on how to structure this. Aim for little code change and consistent with what's being done in the 2 other pipelines I have pointed out and it should be a good starting point.
Forcing to use `stride` on this to prevent handling conflicts at the boundaries, would certainly help (A bit similar to what's being done here I would say: https://huggingface.co/blog/asr-chunking)

If you start working on this, don't hesitate to share your work early and ping me or others to receive feedback on the direction.

Thanks for the advice and links.
I see, I think that's where I've been going wrong. Ultimately, I want to be able to:
- make one call to `tokenize`, passing the entire batch at once
- Then apply batching before passing inputs to the `forward` function
- Then postprocess all inputs at once, such that the `overflow_to_sample_mapping` values are all correct and batch-independent (unaligned, as you say)

From the sounds of things, that's where the `collate_fn` and `uncollate_fn` functions come in. Does that seem like a sensible workflow?

> make one call to tokenize, passing the entire batch at once
    Then apply batching before passing inputs to the forward function
    Then postprocess all inputs at once, such that the overflow_to_sample_mapping values are all correct and batch-independent (unaligned, as you say)
    
This is exacly what `ChunkPipeline` does for you.. for essentially free.
You need to yield items in the preprocess instead of of returning a whole batch, and add `is_last` key (so the unbatcher knows when to stop). `is_last` just needs to be True for the last item of the overflowing tokens.

`_forward` will see the batch, but no modification is needed in this function, since it's the class itself that will handle the `collate_fn` again for free for you. The batch_size might be very different than the number of items `preprocess` sent, it's very orthogonal. So you could have a large input that gets split into 100 chunks, and still process them with `batch_size=64` if that's the capacity limit of your GPU let's say.

The difference between `batch_size` and chunk_number is essential and often overlooked :) 

`postprocess` will see all the items at once (the same number of items `preprocess` sent.




Great, that sounds perfect then (I should have looked at that before, it's been... difficult trying to hack that into a standard `Pipeline` subclass).

If I come up with anything useful and reusable, I'll open a PR

> Great, that sounds perfect then (I should have looked at that before, it's been... difficult trying to hack that into a standard Pipeline subclass).

Perfectly understandable `ChunkPipeline` are relatively new, and mostly introduced after a big refactor, only to enable batching back onto `ZeroShot` and `QuestionAnswering`. It's not super advertised, since implementation is definitely more involved.

Hi all, I have come to a similar issue. My setup is I get the text of an image caption with several panels on it and I must separate the panels. 

My approach so far is using NER to classify tokens as 'O' or 'START-PANEL'. 

We use a BERT-based model but a substantial part of the captions we have are >512 tokens. I would be glad to give a try to build a kind of LongTextTokenClassificationPipeline following your comments if this has not yet been tried.

After experimenting with our model, in most of our cases `stride` would not be needed since the model keeps a high performance, but I could try to add something like that if the experience of buildiing this pipeline would be successful and useful for anyone


> We use a BERT-based model but a substantial part of the captions we have are >512 tokens. I would be glad to give a try to build a kind of LongTextTokenClassificationPipeline following your comments if this has not yet been tried.

That would be awesome !

If you want this to be merged in `transformers` there is one key aspect, is that the chunking MUST be opt-in (because it will affect performance on models different from yours so we cannot do it without the users activating the option).

If you want to be easier, you could implement it exactly as you want and share it, and we could check what/how we can merge back afterwards.

Thanks for the answer :) 

I am following an approach that is basically using the `TokenClassificationPipeline` and modify it to be used with `ChunkPipeline` instead of `Pipeline`. 

So far things seem to be working. I am using the `stride` option for generalization. Will deal with how to aggregate the data when arriving to the `postprocess` part, but in the `preprocess` it does not generate a big issue. I am currently in the `preprocess` method. 

My current approach is to first use `_get_sentence_chunks` to estimate the total amount of chunks a sentence will be split on.
The next step is to divide the model inputs into these chunks. Here I am struggling a bit on what would be the most efficient way.

My thought was to set up `return_tensors=None` in the call to the tokenizer. This would give me lists that could be then sliced into the different parts of the sentence. Then, I need to add [CLS] and [END] to each chunk. The other options would be to return the tensors from the tokenizer and then generate a function that slides the tensors and concatenates them with [CLS] and [END], doing it for each framework.

Then I would do a function (still not implemented here) that takes care of converting each list into a vector based on `self.framework` to make the pipeline general. 

Does this sound like a proper approach to you? 


Sorry for the radio silence @Narsil - I actually implemented this back in March, but due to issues with my employer's open source policy, couldn't publish the changes. These issues have been remedied, and I've got a branch with this token classification pipeline and associated tests.

I've implemented the pipeline, but had to migrate the tests from pytest to unittest tests so these tests don't all pass in the HF suite yet. I can open a (draft) PR at this stage to give you chance to comment? I'll try to finish those other test fixes in the next few days and publish the actual PR

Sure, please publish ! 
Better to start the discussion early.

I don't think we hardly rely on `pytest` but I could be wrong.

I've gotten around to updating the test cases on the PR, they're passing now so I've published it. Let me know if it needs changes or explanation

Any updates on this topic? 
Having a mechanism to handle more than 512 tokens for token classification inference is certainly a useful feature to have.

I've got a [PR open](https://github.com/huggingface/transformers/pull/19735) which should solve this, but hasn't been reviewed yet. I've also had some issues with the CI tasks that I need someone from the HF team to help with