Add possibility to evaluate predefined completions based on logits softmax
Author: kevinpl07
Pull request: yes

# What does this PR do?

This PR enables users to evaluate a predefined set of completions for a given prompt based on the softmax probabilities.
I think sometimes you don't want to have completely "free" text generation but rather have a set of possible completions and have the model decide which fits the best. Of course this is not as easy as it sounds but using the mean or the median of the probabilities can be a good estimate of the fitness of the completion.

The function evaluate_completions() does exactly that given the following params:
- input_ids
- completion_list

Usage:
```python
prob_list = model.evalute_completetions(input_ids, completion_tokenized)
```

Example:
Here is a short example for a given prompt and three possible completions.
Code
```python
prompt = "In recent events the collaboration between Albert Einstein and Yann LeCun started getting traction."
completion_options = [" That's why they kissed.\n", " Einstein suggested to work together on Artifical Intelligence.\n", " Kevin is my name\n"]

completion_tokenized = [tokenizer(completion_str, return_tensors="pt").input_ids[0].to("cuda:0") for completion_str in completion_options]
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
input_ids = input_ids.to("cuda:0")

prob_list = model.evalute_completetions(input_ids, completion_tokenized)
for idx, prob_item in enumerate(prob_list):
    print(completion_options[idx])
    print(prob_item)
    print("Mean: " + str(np.mean(prob_item)))
    print("Median: " + str(np.median(prob_item)))
    print("------------------")
```
Results
```
 That's why they kissed.

[0.0034647872671484947, 0.04622786119580269, 0.07665219902992249, 0.017973434180021286, 3.707055611812393e-06, 0.09081998467445374, 0.32161933183670044]
Mean: 0.07953732931995157
Median: 0.04622786119580269
------------------
 Einstein suggested to work together on Artifical Intelligence.

[0.004310959950089455, 0.0010098001221194863, 0.15259899199008942, 0.005570804700255394, 0.3281031548976898, 0.44232818484306335, 0.00010287049371981993, 0.5048327445983887, 0.9722456932067871, 0.8074969053268433, 0.2874982953071594, 0.09513897448778152]
Mean: 0.30010311499366554
Median: 0.22004864364862442
------------------
 Kevin is my name

[5.61575961910421e-06, 0.005230772774666548, 0.0016122044762596488, 0.008357840590178967, 0.009741517715156078]
Mean: 0.004989590263176069
Median: 0.005230772774666548
------------------
```
We see that by using the mean or median, we can distinguish between a good and a bad fit.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@patrickvonplaten


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@patrickvonplaten @sgugger 

Hey @kevinpl07,

Thanks for the PR. This use case sounds quite specific to me and I don't think core transformers is the right place for it especially since it's really just a helper method IMO. It'll be pretty hard to maintain this method I think.

Since this is a bit of an advanced use case, we can assume that users could write this method on their own I think:

```
from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained("gpt2")

bos_token_id = bos_token_id if bos_token_id is not None else self.config.bos_token_id
inputs_tensor, _, model_kwargs = model._prepare_model_inputs(inputs, bos_token_id, model_kwargs)
batch_size = inputs_tensor.shape[0]

if self.config.is_encoder_decoder:
    input_ids = model._prepare_decoder_input_ids_for_generation(
        batch_size,
        decoder_start_token_id=decoder_start_token_id,
        bos_token_id=bos_token_id,
        model_kwargs=model_kwargs,
    )
else:
    input_ids = inputs_tensor

orig_input_ids = input_ids.clone()
complete_result = []

for completion in completion_list:
    input_ids = orig_input_ids.clone()
    result = []
    for completion_token in completion:

        model_inputs = model.prepare_inputs_for_generation(input_ids, **model_kwargs)
        # forward pass to get probabilities
        outputs = model(
            **model_inputs,
            return_dict=True,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
        )

        next_token_logits = outputs.logits[:, -1, :]
        probs = nn.functional.softmax(next_token_logits, dim=-1)
        result.append(probs[0][completion_token].item())
        input_ids = torch.cat([input_ids, completion_token.view(1, -1)], dim=-1)
        model_kwargs = model._update_model_kwargs_for_generation(
            outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
        )
    complete_result.append(result)
```

What I think what could be very useful is to write a blog post or a forum post about this - do you think this might be interesting for you? :-)

Thanks for your review @patrickvonplaten. It is very specific and I totally understand your thought process :)
 - closing the PR
 - 
Any place you could recommend for such a blog post?

Thanks for the understanding @kevinpl07! 

Sure! If you prefer to make it a short blog post, you could do a forum post: https://discuss.huggingface.co/ like we've done it here e.g.: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175 

If you prefer to make a longer blog post we could also add it here: https://huggingface.co/blog (happy to help!)