Adding `batch_size` support for (almost) all pipelines
Author: Narsil
Pull request: yes

# What does this PR do?

When running pipeline on a dataset, with a small model (relative to the GPU). It can be good to be able to batch
the `forward` pass for performance.

This PR addresses this by adding `batch_size` argument.

This PR contains


- [x] Some facilities for batching and unbatching, not handled within each individual pipelines
- [x] Automated testing for ALL small models + pipelines of this functionality
- [x] Disabled for `question-answering` and `zero-shot-classification`. They are trickier because they already use batching with candidate labels and question features. The full solution would involve moving the iterator to real N [hypothesis, template] and batching there, and having another iterator on top that recreates the current `zero-shot`/`question-answering` results. Should we add that capabilities, at least for these 2 pipelines we would have a much better idea of alignement.
- [x] Ran all slow (pipelines) tests without issue
- [x] Refactor the batch/unbatch for better quality code
- [x] More doc, caveats about this argument and use cases, benchmarks and so on.
- [ ] Need to think about TF which has currently no support (both streaming and batching)


The good example (https://gist.github.com/Narsil/4e1c36d7cf8477e5c1d580585860810e):

This code was executed on GTX 970 (and Titan RTX with similar conclusions), model is `distilbert-base-uncased-finetuned-sst-2-english` (250Mo bin file)

The old pipelines GPU method of iteration is excluded because it's an order of magnitude slower in all cases.
```
------------------------------
Streaming no batching
100%|██████████████████████████████████████████████████████████████████████| 5000/5000 [00:26<00:00, 187.52it/s]
------------------------------
Streaming batch_size=8
100%|█████████████████████████████████████████████████████████████████████| 5000/5000 [00:04<00:00, 1205.95it/s]
------------------------------
Streaming batch_size=64
100%|█████████████████████████████████████████████████████████████████████| 5000/5000 [00:02<00:00, 2478.24it/s]
------------------------------
Streaming batch_size=256
100%|█████████████████████████████████████████████████████████████████████| 5000/5000 [00:01<00:00, 2554.43it/s]
(diminishing returns)
```

**This seems promising !**

However, this has:
- Perfect alignment (all inputs are exactly the same length)
- Small model (lots of GPU RAM left for inputs and intermediary results)

Let's look at another example, which might (or not) be a bit more realistic:
Using varying size inputs (https://gist.github.com/Narsil/de88b2d7c242c29772a61af56a5c8270)

```
------------------------------
Streaming no batching
100%|█████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:30<00:00, 32.51it/s]
------------------------------
Streaming batch_size=8
100%|█████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:29<00:00, 33.62it/s]
------------------------------
Streaming batch_size=64
100%|█████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:29<00:00, 34.29it/s]
------------------------------
Streaming batch_size=256
  0%|                                                                                                                                          | 0/1000 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/home/nicolas/src/transformers/test.py", line 38, in <module>
    for out in tqdm.tqdm(pipe(dataset, batch_size=256), total=len(dataset)):
  File "/home/nicolas/src/transformers/.venv/lib/python3.9/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
....
    hidden_states = self.intermediate_act_fn(hidden_states)
  File "/home/nicolas/src/transformers/.venv/lib/python3.9/site-packages/torch/nn/functional.py", line 1555, in gelu
    return torch._C._nn.gelu(input)
RuntimeError: CUDA out of memory. Tried to allocate 472.00 MiB (GPU 0; 3.95 GiB total capacity; 2.13 GiB already allocated; 266.75 MiB free; 2.49 GiB reserved in total by PyTorch)
```

Here we can see, no speedup was achieved, and we actually crashed for large batch size.
This is entirely due to non alignment.

The problem can even be made worse, when you have large batch sizes, and RARE very long sentences (https://gist.github.com/Narsil/357519fd385d864bfec3caf5aa8df575).

```
------------------------------
Streaming no batching
100%|█████████████████████████████████████████████████████████████████████| 1000/1000 [00:05<00:00, 183.69it/s]
------------------------------
Streaming batch_size=8
100%|█████████████████████████████████████████████████████████████████████| 1000/1000 [00:03<00:00, 265.74it/s]
------------------------------
Streaming batch_size=64
100%|██████████████████████████████████████████████████████████████████████| 1000/1000 [00:26<00:00, 37.80it/s]
------------------------------
Streaming batch_size=256
  0%|                                                                                 | 0/1000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/nicolas/src/transformers/test.py", line 42, in <module>
    for out in tqdm.tqdm(pipe(dataset, batch_size=256), total=len(dataset)):
....
    q = q / math.sqrt(dim_per_head)  # (bs, n_heads, q_length, dim_per_head)
RuntimeError: CUDA out of memory. Tried to allocate 376.00 MiB (GPU 0; 3.95 GiB total capacity; 1.72 GiB already allocated; 354.88 MiB free; 2.46 GiB reserved in total by PyTorch)
```

Here we are actually 5x SLOWER on the batch_size=64 than on the non batched version. That is because the rare long sentence is so long, it actually forces the whole batch to be pad to its sequence length, and use much more memory and processing power (the padding tokens ARE processed by the GPU, they just don't influence the end result).

For users, a rule of thumb is:

- Measure performance on your load, with your hardware. Measure, measure, and keep measuring. Real numbers are the only way to go.
- If you are latency constrained (live product doing inference), don't batch
- If you are using CPU, don't batch.
- If you are using throughput (you want to run your model on a bunch of static data), on GPU, then:
  - If you have no clue about the size of the sequence_length ("natural" data), by default don't batch, measure and try tentatively to add it, add OOM checks to recover when it will fail (and it will at some point if you don't control the sequence_length.)
  - If your sequence_length is super regular, then batching is more likely to be VERY interesting, measure and push it until you get OOMs.
  - The larger the GPU the more likely batching is going to be more interesting
- As soon as you enable batching, make sure you can handle OOMs nicely.

There are no good (general) solutions for this problem, and your mileage may vary depending on your use cases. Which is why for now:


- batch_size=1 by default (both for speed and OOM, issues we can't guess the correct parameters, at least with batch_size=1 we have the smallest chance possible to go OOM).
- batch_size = 1 is somehow comparable in speed to batched data with irregular data sizes (which is an important use case, like live products where latency also matters).
- Other batch_sizes are opt-in, because it might be valuable for users to use it (for instance when checking some metric on some dataset which has very regular input lengths, but then it's a user responsibility to check for OOM and slowness).
- batch_size > 1 won't work for `tokenizer`/`feature_processor` that don't have a padding mecanism (if they require it).


It would be ideal if `pipelines` could start taking that responsibility on its shoulders and start batching dynamically for users but it's a hard problem right now:

- It's hard to evaluate OOM, and OOM might happen late (so batch_size will always to have to be somehow dynamic during the streaming process)
- It's even harder to evaluate the slowness factor due to padding, `pipelines` would have to count them, do some kind of batch exclusion mecanism.
- Padding issue could be helped quite a bit with RaggedTensors, however, they also don't play that nicely with the GPU capabilities (which need as much aligned/regular data as possible.


Some other links/issues/discussions:

https://github.com/huggingface/transformers/pull/11251
https://discuss.huggingface.co/t/how-to-change-the-batch-size-in-a-pipeline/8738
https://discuss.huggingface.co/t/how-to-make-pipeline-automatically-scale/7432
https://github.com/huggingface/transformers/issues/13141
https://github.com/huggingface/transformers/issues/12195
https://gist.github.com/Narsil/ee5c09875e74fa6f018dc6d014f6c06c

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@LysandreJik @sgugger 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
Release done, merging.