ENH: added new output_logits option to generate function
Author: mbaak
Pull request: yes

# What does this PR do?

output_logits option behaves like output_scores, but returns the raw, unprocessed prediction logit scores, ie. the values before they undergo logit processing and/or warping. The latter happens by default for the regular output scores.

It's useful to have the unprocessed logit scores in certain circumstances. For example, unprocessed logit scores are very useful with causallm models when one wants to determine the probability of a certain answer, e.g. when asking a question with a yes/no answer. In that case getting the next-token probabilities of both "yes" and "no" (and/or their relative ratio) is of interest for classification. The reason for getting these _before_ logit processing and/or warping is b/c a) that can change the probabilities or b) reject the tokens of interest / reduce the number of tokens to just 1.

In practice this can be used to generate confidence / classification scores when eg. using causallm models for question-answering tasks. Query your language model with: "Is the {statement} correct? Answer yes or no:", take the raw logit scores and softmax them, and calculate the score: prob(yes) / (prob(yes) + prob(no)) to get a useful classification score. 

For an example use-case see paper TabLLM: Few-shot Classification of Tabular Data with Large Language Models by Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. https://arxiv.org/abs/2210.10723

In addition:
- added dedicated unit test: tests/generation/test_utils/test_return_unprocessed_logit_scores which tests return of logics with output_logits=True in generation.
- set output_logits=True in all other generation unit tests, that also have output_scores=True.

Fixes # (issue)
NA

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
      (Yes, I've seen it discussed but now cannot refind the link.) 
- [ X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ X ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- generate: @gante
- text models: @ArthurZucker and @younesbelkada


Comments:
Hi @mbaak 👋  Thank you for opening the PR 🤗 

I haven't seen a request for this feature (or I have no recollection of it). Personally, I am also not very convinced that it is very useful -- at the end of the day, the model will select the next token _after_ applying the logits processors, if there are any. Since merging this PR would increase the complexity of our codebase AND I'm not convinced, I won't be merging it for now.

However, I may have a poor perception of this feature's usefulness. I'm going to do my standard bargain: if this comment gets 20 reactions, it means there are users looking for this feature, in which case I will add the feature 🤗 (Whoever does the 20th reaction, please tag me)

Hi @gante,

Let me try to explain the use-case better, my description was a bit short and perhaps unclear. 
(Indeed for next-token-generation the raw logits are not relevant.) 

I'm using a RAG setup for question-answering on large documents. (For example, a document describes a project, and a question can be: what's the location of the project?) As causallm model I'm using llama2 (13B) with default settings, meaning with logit processing and warping turned on. The answers are generated as usual, so based on the processed logits to generate the best possible next tokens.

However, these answers need to be reviewed for mistakes/hallucinations. So we want a confidence score for each answer - one that helps in the review. A causallm model does not provide that of course, just generated tokens, so we got to be creative and came up with a possible solution. 

(We find this setup with an LLM is significantly more accurate than a RAG using dedicated bert-based QA models, which do provide scores.) 
(One can prompt a causallm model to generate a confidence score, but it seems that only works for O(100)B parameter models, not for smaller ones.)

We do the following: 
After each answer the same instantiated model is queried again and asked: "Given the context is the provided answer correct? Yes or No." We are then interested in the relative probability: P(Yes|text) / (P(No|text) + P(Yes|text)). 
Interestingly, and perhaps surprisingly, in practice this ratio turns out to be a pretty reliable confidence score, ie. closer to zero is more inaccurate and closer to one is more accurate. 

But for this to work we need the unprocessed logits. B/c after warping (and using this query) normally only one token remains, yes or no, which has the (renormalized) probability 1, so the confidence score is always 0 or 1. That does not help us.

So I'm using a causallm model in a somewhat unconventional way. But one that I believe is very useful, in the sense that it provides a (much needed!) functionality that is otherwise missing for RAGs (as far as I have seen).

Hope this makes it more clear! 🙂

There may be a different way to do this, but I'm afraid I am not aware of it. Hence the PR for unprocessed logits, which for the reason above I think this would a useful addition to have.  

@gante This feature was requested earlier: https://github.com/huggingface/transformers/issues/17521

@sbrugman Thanks for the reminder. 
@gante Does that change you mind? :-) (In the other thread you are okay with exactly this.) 

This would be very beneficial for my current project, I would love this PR to be merged. 

upvote

It seems I was wrong, and several people do want it :) Reverting my decision, I'll review the PR so as to include the feature.

@mbaak indeed, with sampling the `top_k` argument (active by default) erases a substantial part of the logits signal.

@vwxyzjn given your comment in the other issue, this might be useful to you :)

Thanks for the comments! I'll pick it up.

@gante I've implemented your feedback, the code should be good to go I think!

@mbaak To make our CI go green, you will need to:
1. rebase with the latest `main`
2. run `make fixup` on your terminal, within the transformers folder
3. force push the changes

@gante Done!

@mbaak there seems a missing `output_logits` somewhere in the code, CI is complaining :D

@gante Fyi I'm working on fixing the tests. (By default they're skipped locally, that's why I missed them earlier.)

@gante I fixed the CI tests (I had only run the generation/ tests locally, not the model ones), and have implemented @amyeroberts' feedback. I think it's good to go now!

@mbaak thank you for iterating and making the library richer 💛