Add document token classification pipeline (#1)
Author: vaishak2future
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution 

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adds Pipeline for Document Token Classification. Code is mostly based on PR for Document Question Answering. https://github.com/huggingface/transformers/pull/18414


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

@Narsil

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts and @NielsRogge
- speech models: @sanchit-gandhi

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger and @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->
@Narsil 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21012). All of your documentation changes will be reflected on that endpoint.

Hi @vaishak2future 

Did you know that layoutlm already implements `object-detection` : https://huggingface.co/Narsil/layoutlmv3-finetuned-funsd

This might be close enough to this, no ?

@Narsil , thank you for looking at the PR. While Object Detection does solve this particular instance of the problem, we see Document Token Classification as a multimodal task separate from the unimodal task of Object Detection. Document Token Classification requires two modalities - an image and a set of tokens.

This gives control to the user to use their OCR of choice (especially for languages that are not well handled by Tesseract), but also to choose their own tokens that might not be text on the image itself. 

@Narsil All checks are now passing. Could you please review? Thanks.

Hi @vaishak2future ,

I understand the ideas to remove the Tesseract where needed. For the extra tokens, where you imagining extracting tokens from PDF directly maybe ? (This was also an idea behind `document-question-answering` where the idea is that we could always fuse the pipeline later with regular `visual-question-answering`).

Here there are a few things that make me hesitant:

- Pipelines are made to be usable by non ML programmers, here, it's kind of tricky since tokens and boxes and such are quite ML involved
- Pipelines are made to be relatively generic over different model types, here only layoutlm would work as-is. The idea is to keep the number of pipelines relatively small, so discoverable by users.

That being said, enabling power users like your use case should be supported IMO. I would have to look at how to implement within `object-detection`. But I don't see any issue with adding extra parameters for such niche, but extremely useful use-cases.
For instance `asr` pipeline enables users to send the raw audio frames directly which IMO is seemingly the same idea (bypass or modify very specifically some preprocessing which would be the OCR in your case)

What do you think ?

Pinging @sgugger @LysandreJik for other opinions on this.

Regardless, I briefly looked at the PR, the code seems good, there are a few nits regarding how tests are structured and how many different inputs are accepted, but overall it looks quite good. I'll delay my comments after we reach a decision on this as there's no big structural blockers on my end imo.

This looks very specific to one model. We can't host all possible pipelines in Transformers, so in such a case, we should rely on the code on the Hub for pipeline feature. You can see pointers [here](https://huggingface.co/docs/transformers/add_new_pipeline#share-your-pipeline-on-the-hub).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.