'return_dict_in_generate' and 'output_scores' argument in BartForConditionalGeneration.generate()
Author: ktr0921
Pull request: no

Dear all

I was wondering if I can ask you some questions about how to use `.generate()` for BART or other pre-trained models. The example code is,
```
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
path = 'facebook/bart-large'\
model = BartForConditionalGeneration.from_pretrained(path)
tokenizer = BartTokenizer.from_pretrained(path)

ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')

# Generate Summary
summary_ids = model.generate(
    inputs['input_ids'],
    num_beams=4, num_return_sequences=2, max_length=5, early_stopping=True,
    output_scores=True, return_dict_in_generate=True,
)
print(summary_ids.keys())
print(summary_ids['sequences'])
print(summary_ids['sequences_scores'])
print(len(summary_ids['scores'][0]))
print(summary_ids['scores'][0].size())
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False)
       for g in summary_ids['sequences']])
```

Then, the output is,
```
odict_keys(['sequences', 'sequences_scores', 'scores'])
tensor([[   2, 2387, 2387,  964,    2],
        [   2, 2387,    4,    4,    2]])
tensor([-0.8599, -0.9924])
4
torch.Size([4, 50265])
['MyMy friends', 'My..']
```

Do not worry about poor performance, ['MyMy friends', 'My..'], since I am only trying to understand how this works. So, the question is,
1. `return_dict_in_generate=True` returns `['sequences']`, but together with `output_scores=True`, it returns `['sequences', 'sequences_scores', 'scores']`. There are other arguments, like `output_attentions` or `output_hidden_states`. [BART BartForConditionalGeneration documents](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration) do not explain anything about `.generate()`. So, I searched further and found [Utilities for Generation](https://huggingface.co/transformers/internal/generation_utils.html?highlight=generate) that seems to talk about generating outputs using `.generate()` and [Huggingface transformers model](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.generate) that seems to talk about the general methods of base classes, PreTrainedModel, but there is no document that shows what each variable, ['sequences', 'sequences_scores', 'scores'], actually work or how they are computed. Where is the documents for this?
2. Is `sequences_scores` computed as, \sum_{t} \log p(y_{t} | x, y_{t<})?
3. How do you get `sequences_scores` from `scores`? My initial guess was to apply softmax on `scores` in `dim=1`, then get `topk` with `k=1`, but this does not give me very weird answer.
```
import torch

sm = torch.nn.functional.softmax(summary_ids['scores'][0], dim=1)
topk = sm.topk(k=1, dim=1)
print(sm)
print(topk)
print(summary_ids['sequences'][0])
```
which comes out as
```
tensor([[1.2851e-04, 8.8341e-12, 2.4085e-06,  ..., 3.9426e-12, 2.8815e-12,
         1.0564e-08],
        [1.9899e-05, 1.9899e-05, 1.9899e-05,  ..., 1.9899e-05, 1.9899e-05,
         1.9899e-05],
        [1.9899e-05, 1.9899e-05, 1.9899e-05,  ..., 1.9899e-05, 1.9899e-05,
         1.9899e-05],
        [1.9899e-05, 1.9899e-05, 1.9899e-05,  ..., 1.9899e-05, 1.9899e-05,
         1.9899e-05]])
torch.return_types.topk(
values=tensor([[9.9271e-01],
        [1.9899e-05],
        [1.9899e-05],
        [1.9899e-05]]),
indices=tensor([[2387],
        [   0],
        [   0],
        [   0]]))
tensor([   2, 2387, 2387,  964,    2])
```
First token 2387 appears to be correct, but from the second, the probability is 1.9899e-05, which is just equivalent to 1/len(tokenizer). This seems to me that all the tokens are likely to be generated equally. So, How do you get `sequences_scores` from `scores`?
4. How do I get the probability of all the conditional probability of output tokens? For example, if `.generate()` gives output as [I, am, student], then how do I get the conditional probability of each token? [Pr(I | x), Pr(am | x, I), Pr(student | x, I, am)]. Initially, I thought it was 'scores', but I am not sure now.
5. Since I find it difficult to find documents/information on `.generate()` nor any information above, is this something that experienced researchers in NLP or programming would just be able to guess? 

Thank you in advance

Comments:
hi @ktr0921 

Please use the [forum](https://discuss.huggingface.co/) to ask such question, [this](https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175) thread might help