Ranking a pre-defined list of output candidates for LMs
Author: yuchenlin
Pull request: no

### Feature request

Can we support a feature that given an input and a list of output candidates for a LM (say fine-tuned T5 or GPT-2), we can get a score for each output candidate and then return the top K of them?

For example, I fine-tuned a seq2seq model (e.g., BART or T5) for summarization, and now given an input doc and a list of candidate summaries, I want to know which ones are the best. I do not want the fine-tuned model to decode and generate its summaries as we normally do with them. 

I believe we can do this by computing the token-by-token log-likelihood (with or without length normalization), and then return the top K from the candidate list.


### Motivation

I was reading the paper of T0, and it mentioned that they used such a method to evaluate tasks with multiple choices:



The paper describes it like this on Page 6 from https://arxiv.org/pdf/2110.08207.pdf:

> For tasks that involve choosing the correct completion from several options (e.g. multiple choice
question answering), we follow Brown et al. (2020) and use **rank classification** to evaluate our
model: we compute the log-likelihood of each of the target options under the fine-tuned model and
select the option with the highest log-likelihood as the prediction. For simplicity, we do not apply
length normalization to the log-likelihoods of the target options.

### Your contribution

I found their code that maybe helpful but I feel that they are a bit hard to use directly. 

https://github.com/bigscience-workshop/t-zero/blob/25c0761427f3894a8ec5a062a075b96037fb1492/t0/model.py#L67

Comments:
Hey @gante , I was wondering you may will also have great suggestions on this. Thanks a lot in advance! :D. 

Hey @yuchenlin 👋 We usually don't add those sorts of tools to `transformers`, but I'd be happy to guide you.

First of all, I'd suggest to check [the documentation and the examples for this function](https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/text_generation#transformers.GenerationMixin.compute_transition_scores). Then, by invoking it for several models and/or inputs, you will be able to re-rank candidate options according to your needs :)

If you find the end results satisfying, I'd suggest opening a Spaces so it can be shared with the world 🌍 

Hi @gante thank you so much for the help! I was reading this before but it seems that this method can only output the transition scores for tokens that are considered by the model.generate process. However, the more general case for this application is that we have some external output candidates that may not be generated by "model.generate()". 

And sure I will put a solution to a Spaces if I manage to do this and I believe this will help many others! :D 

I see. We have in our plans to build a function that returns the score for any candidate sequence, but have other competing priorities :) Feel free to have a go at it, if you're interested!

Hi @gante , thanks for letting me know. 

I managed to develop a solution here: https://github.com/yuchenlin/rank_outputs/blob/main/rank_outputs/main.py 

Will try to wrap it up as a more general tool. 

Thanks!