[Large PR] Entire rework of pipelines.
Author: Narsil
Pull request: yes

# What does this PR do?

tl;dr: Make pipeline code much more consistent and enable large speedups with GPU inference.

# GPU pipeline

Currently the way pipeline are setup, it's kind of hard to keep the GPU busy 100% because we're not enabling the use of DataLoader (on pytorch), which is necessary to keep CPU working on next items to tokenize, while processing an item on GPU.

We cannot realistically use the current API to maximize utilization:

```python
for item in dataset:
    # item == "This is some test" for instance
    output = pipe(item)   
    # output == {"label": "POSITIVE", "score": 0,99}
```

So we need to change up the API to something closer to what `DataLoader` does, which is use an iterable, which enables to have worker CPU threads process next items while the GPU is busy on the current one, meaning we're now using 100% of the GPU.

```python
for output in pipe(dataset):
    # output == {"label": "POSITIVE", "score": 0,99}
    pass
```

In order to make that change possible, we **need** to separate better what happens on the CPU vs the GPU.
The proposed way is to split the __call__ of pipeline into 3 distinct function calls

- `preprocess`: in charge of taking the original pipeline input, and output a dict of everything necessary to do `model(**model_inputs)` for instance (or a `generate` call, but stuff that will really involve the GPU.
- `forward`:  In most cases it's a simple function call to the model forward method, but can be more complex depending on the pipeline. It needs to be separate from the other 2 because this is where the GPU might be used. so we can encapsulate more logic around this in the base class (`no_grad`, sending and retrieving tensors to/from GPU etc..)
- `postprocess`: Usually links to processing the logits into something more user-friendly for the task at hand, again usually pretty fast and should happen on CPU (but should be so fast it does not matter really to have a separate thread for this).

In order to increase consistency across pipelines, ALL pipelines will have to implement the 3 methods, and should have a `__call__` method (with exceptions discussed in consistency).
They should be readable on their own too, meaning, the outputs of `preprocess` should be **exactly** what is sent to `forward` and what is returned by `forward` exactly the inputs of `preprocess`. So:

```python
model_inputs = pipe.preprocess(item)
model_outputs = pipe.forward(item)
outputs = pipe.postprocess(model_outputs)
```

will always be perfectly valid, even if not the most efficient.


# Consistency of pipelines

Right now, pipelines are quite inconsistent in their returned outputs.

- Some have parameters to change the output format (this is fine)
- Most pipelines accept lists of items, and will return a list of outputs but:
   - Some will return a single item only if the input was a list of a single item (regardless of what the inputs originally was)
   - Some will do it better and return single item only if single item was sent
- Some will use lists as batching, some will not, leading to slowdowns at best, OOM errors on large lists, and overall pretty poor efficiency on GPU (more info: https://github.com/huggingface/transformers/issues/13141, https://github.com/huggingface/transformers/pull/11251, https://github.com/huggingface/transformers/pull/11251)

Batching on GPU seems like what is speeding up, things, but really it's not at inference times, batching in ML is used because of gradients and it's necessary for the gradient descent to be smooth, the speed part of the GPU is really linked to overall GPU usage, using `DataLoader` is the key part here. Nonetheless, sometimes, depending on actual hardware, pipeline, and input data, batching *can* be used efficiently, so the new design should enable that. However, it shouldn't be done the way it's currently setup, which is some pipelines do, some don't and no consistency overall, it should be done on a different layer than dataprocessing part of the pipeline.

Because of the inconsitencies mentionned above, this refactor will include some `__call__` methods to change the return type based on what was previously there so (`preprocess`, `forward` and `postprocess` are mostly pure, while `__call__` will handle backwards compatibilty)

# Parameter handling

Another cause of concern for pipelines was parameter handling. Most parameters were sent to `__call__` method, but some where sent to `__init__`. Some in both.

That meant that you would have to look though the docs to guess if you needed to to 

```python
pipe = pipeline(....., num_beams=2)
outputs = pipe(item)
# or
pipe = pipeline(....)
outputs = pipe(item, num_beams=2)
```
The goal in this PR, was to make that explicit, so BOTH will be supported and have the exact same behavior.
In order to do that, we introduced a new mandatory method `set_parameters` which would be called both in `__call__` and `__init__` in the same way so that it would always work.

1. Because this new `set_parameters` is a standard method, we can use it to properly discard unexpected keyword with a real errors instead of just ignoring it.
2. Because `__init__` and `__call__` are now base class only (roughly), we can capture parameters much better, meaning we don't have extra layer of parameter guessing (is it tokenization param, model param, pipeline param ?). Each method will capture everything it needs and pass on the rest, the ultimate method in the chain is `set_parameters` which might be specific parameters, or accept everything (like **generate_kwargs, so utlimately `generate` will have the final word).
3. Because `set_parameters` will be called at least 2 times and we don't know which one will have actual real values, it needs to be done in a somewhat odd way. The ways most pipelines will do, is simply have a default argument to `None`, so if the argument is `None` we know that the caller didn't supply this argument so we don't override it (the default one is defined in the `__init__` if dynamic or directly in the class if static. This however does not work when `None` is a valid choice for some parameter, this is true **only** for `zero-shot-classification` test, where we specially test that we raise some error when passing `None` as a value (so it can probably be changed, but will be backward incompatible regarding tests). For those, more complex logic is required.
4. Because we're now using `self` as the holder for parameters that means that using threading mecanisms to run the pipelines might lead to some oddities (but people most likely aren't using 1 pipeline on different threads, most likely shouldn't be at least). Other options are possible but would passing them though all 3 functions `preprocess`, `forward` and `postprocess` reducing readability IMHO, for debattable gains.

# Results

Currently we're sitting here performance wise

bench code
```python
from transformers import pipeline
from transformers.pipelines.base import KeyDataset
import datasets
import tqdm

pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h", device=0)
dataset = datasets.load_dataset("superb", name="asr", split="test")

print("New style of pipeline")
for out in tqdm.tqdm(pipe(KeyDataset(dataset, "file"))):
    pass

print("Old style of pipeline")
for item in tqdm.tqdm(dataset):
    out = pipe(item["file"])
```

Speed (done on old suffering GTX 970):
![F02AXFBCPJN](https://user-images.githubusercontent.com/204321/131305601-e1b75b93-97e6-47c8-a9ce-55bcbbaece58.png)

## Backward compatibility

We're currently sitting at 100% backward compatibility regarding tests. We're not however 100% backward compatible.
By fixing the inconsistencies of pipelines, we will break any code that was using parameters wrong (as they will suddenly start working or crashing because they're invalid).

## Tensorflow

I mentionned `DataLoader` which will be used to great effectiveness on Pytorch + `list` inputs or `Dataset` input. (on single inference on GPU + pt, you will get a warning, prompting you to use more efficient methods)

On tensorflow however, more work is needed to make it faster there too. At the very least we shouldn't degrade performance too much, this has to be checked (both GPU and CPU). Ideally we would have a similar mecanism than `DataLoader` to maximise efficiency on GPU tensorflow.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
What about integrating this [idea](https://github.com/huggingface/transformers/issues/13274) into this rework?

@xegulon It would be a great addition, we already have similar functionnality within HF.

The code is not open source just because it's messy and wouldn't fit `transformers` requirements (backward compatiblity and maintaining this is out of scope in our opinion) but we do reuse most tools that we provide (like export_onnx),  so it's mostly plumbing.

If we can find something clean enough, it's probable it would be a welcome addition.

Few caveats to mention: 
- Using `ONNX` in fully optimized mode makes it hardware dependent (you HAVE to run on similar hardware as where the optimized file was created). 
- Using quantization might lead to performance drop (but also huge speedup).
- Using ONNX with fancy methods like `generate` is much harder to do to keep performance (you have to take care of `past_key_values`).
- Using ONNX with `generate` and running on GPU is actually counterproductive because we can't run the beam search directly on GPU tensors (that's an ORT limitation). So there's a lot of back&forth between GPU and CPU which is bad for performance. (We also tried the `beam_search` proposed by ORT but didn't find it was worth it as implementation differs significantly from transformers.)

With those caveats in mind, feel free to add a PR, it would be a welcome addition if we manage to make it readable and orthogonal (the new refactor should help for sure).
Try to make the PR small and more like PoC so everyone could weigh in in terms of design (most notably transformers core maintainers)

Hey, it's really great to see work on general code organisation to any degree.  Thanks for your work.

It looks like this PR introduced a bug around completing empty prompts:
```
transformers.pipeline('text-generation')('')
```
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 150, in __call__
    return super().__call__(text_inputs, **kwargs)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/pipelines/base.py", line 915, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/pipelines/base.py", line 922, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/pipelines/base.py", line 871, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 162, in _forward
    generated_sequence = self.model.generate(input_ids=input_ids, **generate_kwargs)  # BS x SL
  File "/home/user/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/generation_utils.py", line 1016, in generate
    return self.sample(
  File "/home/user/.local/lib/python3.8/site-packages/transformers/generation_utils.py", line 1529, in sample
    outputs = self(
  File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 949, in forward
    transformer_outputs = self.transformer(
  File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 673, in forward
    input_ids = input_ids.view(-1, input_shape[-1])
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous
```

🤑💪🏻

how to use my own dataset, that is txt file ,per line is the input for NER model
could you pls help me ?


> how to use my own dataset, that is txt file ,per line is the input for NER model
> could you pls help me ?

the example scripts want itin jsonlines or csv. https://huggingface.co/docs/transformers/run_scripts#use-a-custom-dataset  . you can use a tool to convert to jsonlines. it takes some patience to figure out a way to do each step, and then it works.