ChunkPipeline (batch_size enabled on `zero-cls` and `qa` pipelines.
Author: Narsil
Pull request: yes

# What does this PR do?

It enables `batch_size` on the two pipeline where it wasn't possible previously.

The main roadblock for these is that `preprocess` would already return tensors that would contain part of the `batch_size`, meaning we couldn't use `Pipeline.batch_size` independently of the processed data. Both concepts were not orthogonal.

It also means there was no way to limit the size of the batch on a very large input item, by chunking the inputs being fed to the model.


<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

The problem is being solved by changing some assumptions about the format of `Pipeline`, and by creating `ChunkPipeline` (since we didn't want to change ALL pipelines for these two).

This modifications should yield exactly 0 breaking changes, and really just enable the batching for these two pipelines.

They differ from `Pipeline` simply because `preprocess` will not return just one item, but an iterator instead (SquadFeatures for `qa`, and a list of [sentence, hypothesis]  for `zero-shot`).

```python
item = ...
model_inputs = pipe.preprocess(item)
model_outputs = pipe._forward(model_inputs)
outputs = pipe.postprocess(model_outputs)
```

Now becomes:

```python
item = ...
all = []
for model_inputs in pipe.preprocess(item):
    model_outputs = pipe._forward(model_inputs)
    all.append(model_outputs)
output = pipe.postprocess(all)
```
When doing batching this becomes a bit more convoluted since we need to keep track of 2 things, where we are in the batch currently, and do we still need to accumulate in the `all` variable or not.

Everything is handled in iterators but it's equivalent to:

```python
item = ....
batch = []
all_model_outputs = []
for item in all_items:
    for model_inputs in pipe.preprocess(item):
        batch.append(model_inputs)
        if len(batch) != batch_size:
           continue

         model_outputs = pipe._forward(collate(batch))
         batch = []
        for model_output in model_outputs:
             if model_outputs["is_last"]:
                 output = model.postprocess(all_model_outputs)
                 all_model_outputs = []


                 yield output
             else:
                 all_model_outputs.append(model_output)
```

We are doing that using a special `is_last` boolean value that needs to be sent by the `preprocess` function and passed along (as a tensor) by the `_forward` function. That's because it's a bit hard for the iterators to deduce that information on their own if it's not propagated by the `_forward` function. Making it explicit makes reasoning about this clearer as the pipeline is not responsible both for sending it and passing it along (it's still magically accumulated back as a list before sending to `postprocess`).

This could be entirely magically inferred but it feels the complexity incurred by not be worth it (it's already complex enough).

<table>
  <tr>
    <td>
Pipeline
<img src="https://user-images.githubusercontent.com/204321/140971945-5e20a775-c291-4640-8c2e-f349337a4fea.jpeg" width=200>
</td>
<td>
Pipeline + batch_size
<img src="https://user-images.githubusercontent.com/204321/140971950-6ebabf33-6e93-4975-b397-26d640aa6321.jpeg" width=200>
</td>
</tr>
<tr>
<td>
ChunkPipeline
<img src="https://user-images.githubusercontent.com/204321/140971957-e9278c13-0847-4c64-85c9-3e03fbb66e9b.jpeg" width=200>
</td>
<td>
ChunkPipeline + batch_size
<img src="https://user-images.githubusercontent.com/204321/140971955-550ee033-e26e-41f4-b81f-28bb2195879b.jpeg" width=200>
</td>
</tr>
</table>


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
@LysandreJik I split this PR in 2, were 1 holds the ASR part, and this just enables `batch_size` for QA and zero-shot.

It has benefits, but also increases complexity of the overall code. Not 100% sure about this.

Ok thanks, this is a big PR, will review it in the coming days.

@LysandreJik I added a bunch of documentation as comments for now.

When this pass it could be helpful to add stuff in the main documentation (mainly for pipeline creation).


Thanks for adding comments, this looks good. Would you like to merge it now or to include the documentation changes in it before merging?

I want to add some documentation for this now

@LysandreJik I think it's better.

For the documentation, I kept it slightly minimal. Actually I think part of the documentation on `pipeline` belongs to this page https://huggingface.co/docs/transformers/task_summary but it seems that this page talks about a little more than just the pipeline, and also is missing quite a few tasks (and there's also the great work of @merveenoyan on tasks https://github.com/huggingface/huggingface_hub/pull/509 that we might want to link to and/or refer)