[WIP] Adding logprobs to `text-generation` pipeline.
Author: Narsil
Pull request: yes

# What does this PR do?

Given the discussions to add this parameter here is a first (non
working) draft.

- `text-generation` only, we would need to support
  `text2text-generation` too, which unfortunately is a different
  pipeline
- This gets quite complex in the case of `num_return_sequences` +
  `num_beams` to keep track of all the dimensions. There seems
  to be something wrong in the `output_scores` + `num_beams`
- This is IMO quite unaligned with purpose of pipelines.
  Pipelines are supposed to be used for naive non ML practitionners, so
  understanding what tokens are is beyong the purpose of the pipelines.
  I'm creating this PR, just because multiple discussions have been
  started asking for this feature.
- The current code becomes very bloated IMO (and it's not finished to handle all the different params).

`generate` scores already generated a lot of discussion: 
 - https://github.com/huggingface/transformers/issues/17424
 - https://github.com/huggingface/transformers/issues/18942
  
To be clear, it seems everyone involved in said discussions wants parity
with GPT-3 OpenAI and CoHere. since the API is powered by the
pipelines we should implement it here to have the functionnality
within the API.
https://huggingface.co/bigscience/bloom/discussions/89#6322b153a418a789a23f3380
Discussion for Bloom (currently custom code so not concerned, but still to be considered)

Other discussions are internal/email but follow roughly the same pattern as far as I could read.

The purpose of this PR is to gauge interest for this feature and/or
aligning with other APIs.
Please star this PR if you are interested in this feature.
And please do comment if other/more important features are interesting
to you.



<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@spolu @sgugger @gante (Maybe we have some other comments on how to handle TF too.)

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
This looks great. You may want to cap the logprob argument to a small-ish number (maybe 12?) to avoid giving the ability to people to generate massive amount of data from short queries. I think 12 is very fair and covers most use-cases.

Thanks for the PR, but I am very much not in favor of this change. We have developed [tools](https://huggingface.co/docs/transformers/add_new_pipeline#share-your-pipeline-on-the-hub) so that users can put their preferred pipeline code in the repo of their model specifically for use cases like this one. I think those tools should be leveraged instead of changing the base pipeline.

To provide some color, the lack of logprobs for the inference API text generation makes it hardly usable for any non trivial use case.

I also tried deploying a model on inference endpoints and it was not obvious to me where to add code to extract the logprobs?

From a TF standpoint, whatever works for PT should also work there -- their interface should be the same.

Regarding the root issue, it seems that we have to decide whether we want to optimize the default pipelines for a simple interface or for the inference API. I'm in favor of the first, as we have a myriad of solutions for non-trivial use cases:
1. as @sgugger mentioned, custom pipelines can be defined
2. If the user is writing code, replacing the pipeline by a `.generate()` call is also simple (and unlocks several other outputs)
3. if the issue is the GUI for non-trivial use cases, a custom Space can be built (we could create a template where only the model ID needs to be changed, if that would make things easier)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.