Inference API: return token scores for text-generation models
Author: rodrigonogueira4
Pull request: no

# 🚀 Feature request

It would be useful if the Accelerated Inference API returned token scores assigned by Text-generation and Text2text-generation models, similar to what is returned by "Fill mask" models. E.g.:
``` 
output = {
            "sequence": "the answer to the universe is no.",
            "score": 0.1696,
            "token": 2053,
            "token_str": "no",
        },
        {
            "sequence": "the answer to the universe is nothing.",
            "score": 0.0734,
            "token": 2498,
            "token_str": "nothing",
        },
        ...
```

Comments:
cc @Narsil 

Hi @rodrigonogueira4 ,

This is an interesting feature. May I ask what the end goal would be ?
Is it to potentially discard results which are deemed too low ?
Is it to show some kind of confidence (color or otherwise) to your user ?
Any other usage ?

I think clarifying the end usage is really important in this case, since the concept of `score` in `text-generation` is not as straightforward as it may look. If we can pinpoint the core of the issue, it's going to be easier than adding a score without thinking about how it's going to be used.

Elements of reflection:

- Generation, applies between 1 and N (`max_new_tokens` for instance) times the model, meaning we have logits for each `token`, which is not necessarily a word. If the API started returning something like this it would mean we would have to return tokens individually, `[ {"token": "Some", "score": 0.9}, {"token": "##thing", score: 0.99}]` for instance. There are ways to make this work, but not all tokens are readable ([EOS] for instance doesn't print anything, it just means the sequence is ended). Making sure `"".join(tok["token"] for tok in tokens) == original_output` might not be trivial (would have to check, but I don't think we ever provide that guarantee on arbitrary tokenizer, which is why we usually output offsets instead)
- It would need to work, with more complex examples, like `beam_search`, where we return several outputs, each with their own tokens + scores.

The main caveat with that solution I can think of is that as a user, it forces to think about what a token is, and you probably don't want to cut predictions mid-sentence (if you're cutting based on score), so then we need to output words (which not all tokenizers provide). Also low score != bad prediction (The ocean is ["large", "blue"] are both equally valid, meaning probs = [0.5, 0.5] even if the confidence for the model is super high.)

Other option would be to output a single score per generation, but then which score do we return is the product of all probabilities the correct "score" ? This is what is used during beam_search if I am not mistaken, but that number is vanishingly small so using it as a user with a threshold is likely to cause issues as the reference gets smaller as tokens are added. Normalizing this score based on length doesn't seem desirable/feasible ?

Maybe a third option ?

Hi @Narsil,

> This is an interesting feature. May I ask what the end goal would be ?
> Is it to potentially discard results which are deemed too low ?
> Is it to show some kind of confidence (color or otherwise) to your user ?
> Any other usage ?

That's right, these are the use cases I had in mind. In my experience with the GPT-3's API, filtering out outputs using the model's likelihood improves overall results quite a bit.

Returning the probability of the token that was selected by the decoding algorithm (greedy or top-p) at each decoding step is enough to calculate a confidence score for the generated sentence.

Returning scores for beam search would be more complicated. Open AI solved this by returning the top N tokens and their probs at each decoding step.

> The main caveat with that solution I can think of is that as a user, it forces to think about what a token is, and you probably don't want to cut predictions mid-sentence (if you're cutting based on score), so then we need to output words (which not all tokenizers provide). Also low score != bad prediction (The ocean is ["large", "blue"] are both equally valid, meaning probs = [0.5, 0.5] even if the confidence for the model is super high.)

I think that's fine as long as we can compute the perplexity or the likelihood of the entire sequence.

> Other option would be to output a single score per generation, but then which score do we return is the product of all probabilities the correct "score" ? This is what is used during beam_search if I am not mistaken, but that number is vanishingly small so using it as a user with a threshold is likely to cause issues as the reference gets smaller as tokens are added. Normalizing this score based on length doesn't seem desirable/feasible ?

That would work, as long as the user also knows the number of tokens in the generated sequence, so we could normalize by length (by exponentiating to 1/length).




This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I found myself really wanting the same feature:)
Could you please tell me, in the end, it was decided to unnecessarily complicated to implement?:)