How can I access prompt scores/logprobs?
Author: hxiaoyang
Pull request: no

I have tried `model.generate(**inputs, return_dict_in_generate=True, output_scores=True)` but it only gives the scores for generated tokens. For my application, it would be convenient if there’s a similar parameter to `echo` in the OpenAI GPT-3 API that lets us access prompt scores/logprobs, but I have yet to find it. Any help is appreciated!

Comments:
Hey @xiaoyangnickhu 👋 We do have a tool for that, but which is not yet documented: 
👉 [compute_transition_beam_scores](https://github.com/huggingface/transformers/blob/e0b825a8d03f50ed9dbf9fbbbb3b4fcf0b4e4b22/src/transformers/generation_utils.py#L876)

Their arguments should be self explanatory, but let me know if you'd like further guidance :)

Thanks for responding! Some followup questions:
1. Since this function only has access to `scores` for generated tokens, I am having a hard time understanding why the returned `transition_scores` might contain scores for prompt/input tokens. Could you please maybe clarify where this function deals with prompt tokens?
2. Do I need to switch to beam search to use this function? (I have been using greedy decoding.)
3. Also, can I compute prompt token logprobs by applying `log_softmax` to `model(input_ids).logits`?

Thanks!

@xiaoyangnickhu 
1. It does not include the score for the input prompt. The concept of "score" is derived from the assumption in autoregressive language generation where the probability distribution of a word sequence can be decomposed into the product of conditional next-word distributions. The probability of the input tokens is `1` for the input, so we don't include it there.
2. For greedy decoding you'll have to do it manually for now, i.e., gather from the `scores` output the selected token scores (see docs [here](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.generation_utils.GreedySearchDecoderOnlyOutput)). We may include an output for this in the future :)
3. EDIT: Yes you can. 

Thanks! Appreciate the details. Some followups regarding 3:

I see that for each input token `xi`, you feed the sequence `[x1,x2,...,xi]` to `model` to obtain the logits. What is the reasoning behind this? We have been doing `model([x1,x2,...,xn]).logits` (i.e., give `model` the full sequence and apply `log_softmax` to each token); is this the wrong approach? (My goal here is to obtain the vector `[None, logprob(x2|x1), logprob(x3|x1,x2),...,logprob(xn|x1,...,x_{n-1})]`)

Closed the issue by mistake. Sorry...

@gante Would you be able to take a look at my questions above? Thanks!

@xiaoyangnickhu oops, you are absolutely right, you can obtain the conditional logits that way! (I've been so used to work on generate, one token at a time, that forgot that the `.logits` output holds the desired output for all steps).

I've edited my answer above in case someone stumbles across this thread :)

Thanks!!

@gante @hxiaoyang hello, I am using bloom with the API. I need these scores/logprob for input similar to what we can get in OpenAI. Is there a way?

Hey @goelnaman -- by API, what do you mean exactly? I don't think most APIs support it, but I'd be able to tag the right team member :)

Thanks @gante I have tried ...  InferenceApi() and requests.request() but didn't see logprobs of input in any of these.

In OpenAI API, one can get this information by using echo=True, logprobs=... for example.

Hey @goelnaman -- I can confirm that it does not return the scores (API docs [here](https://huggingface.co/docs/api-inference/detailed_parameters#text-generation-task))

The [`text-generation-inference`](https://github.com/huggingface/text-generation-inference) solution also doesn't support it. The only way to get it at the moment is:
1. With local python code, as discussed in this issue
2. With the [Inference Endpoints](https://huggingface.co/inference-endpoints), where you can configure any API.

Sadly, I do not have examples for 2. (it's on my todo list :) )