Returning Confidence Score For Extractive QA Task When Using Non-Pipeline Approach 
Author: UmerTariq1
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

HF's Extract QA pipeline provides an excellent interface for start. It returns 4 values including a **probability score / confidence score**. Unfortunately same is not the case when using the non-pipeline approach i.e using model and tokenizer for question answering. 
[Both methods are mentioned here, The pipeline one and the other](https://huggingface.co/transformers/task_summary.html#extractive-question-answering)

## Motivation
The confidence score will help a lot in various tasks. For example when I am developing a complete pipeline for QA, consisting of recall, retriever and some other models for entity matching and etc. I need to calculate scores of each models and then rank the final list of documents based on the weighted sum of score from each model. I believe this is a very common practice among NLP practitioners and not just for QA task. The point is confidence scores are usually a pretty standard requirement for each model output because we have to take further actions based on its score.

## Your contribution
I want to. but unfortunately I am not at the level where I can understand the code. I have went through the code and I believe its the "decode" function in "QuestionAnsweringPipeline" class which has the code the which generates the probability scores. If you guys can just provide an interface for it or provide docs for how to calculate this score using the model and tokenizer approach then that would be great too. And if you do decide to do this then please also add this addition to docs in the link mentioned at the top.

Thanks. 

Comments:
Correct me if I am wrong but I think the reason why it's not included in the model output is because it's an utility function and not a direct output of the model.

If I am not wrong, calculating the confidence score in a non-pipeline method is straight forward, like how it's done below  
https://github.com/huggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/examples/pytorch/question-answering/utils_qa.py#L151

From the example you shared, adding couple of lines should give us the score.
````
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch
tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
model = AutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
text = r"""
🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose
architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU) and Natural
Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
TensorFlow 2.0 and PyTorch.
"""
questions = [
    "How many pretrained models are available in 🤗 Transformers?",
    "What does 🤗 Transformers provide?",
    "🤗 Transformers provides interoperability between which frameworks?",
]
for question in questions:
    inputs = tokenizer(question, text, add_special_tokens=True, return_tensors="pt")
    input_ids = inputs["input_ids"].tolist()[0]
    outputs = model(**inputs)
    answer_start_scores = outputs.start_logits
    answer_end_scores = outputs.end_logits
    answer_start = torch.argmax(
        answer_start_scores
    )  # Get the most likely beginning of answer with the argmax of the score
    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score
    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))
    # Compute the Score using start_logits and end_logits
    score = outputs.start_logits[0][answer_start] + outputs.end_logits[0][answer_end-1]
    print(f"Question: {question}")
    print(f"Answer: {answer}")
    print(f"Confidence Score: {score}")
````

This also gives us the flexibility to design the confidence score. For example it might also be interesting to boost the confidence score based on intent similarity and entity/NP intersection.

Hope this helps!


Thanks for replying to this @lawliet19189 .  This is good insight.  
I do agree with @UmerTariq1 that there should be an option to return a basic confidence score without jumping through hoops.  At the very least it would save us some time to document that it doesn't exist and a method (like this) that can be used.  That way you don't spend a lot of time searching the documentation for a parameter or method to get them.

So I vote for this feature.