Repository: deepset-ai/haystack
Pull request: Add docu of confidence scores and calibration method
**Proposed changes**:
- Add more detailed documentation of confidence calibration to the reader's usage page.
- Add `calibrate_confidence_scores()` method to reader, which internally calls `eval()` method

closes #1032 

Problem statement:
Simplify usage of confidence scores & add documentation
With # we introduced an option to get more reliable confidence scores for predictions from a QA model / reader. 

So far the usage in Haystack seems not well documented and calibrating scores is not easily accessible. 
Calibration would probably look something like this: 

```
    evaluator_dev = Evaluator(
        data_loader=data_silo.get_data_loader("dev"),
        tasks=data_silo.processor.tasks,
        device=device
    )
    result_dev = evaluator_dev.eval(model, return_preds_and_labels=True, calibrate_conf_scores=True)

```

Or you set it manually to:
```
model.prediction_heads[0].temperature_for_confidence
```

Proposed changes:
1. Let's simplify the calibration by introducing a new "high-level" method in FARMReader (e.g. `calibrate_confidence_scores()`)
2. Let's update documentation so that more users are aware of this functionality (doc strings, but also adding an example either somewhere in the "usage section" or in one of the existing tutorials)

Hints from issue discussion:

Commit message: Explain label "0" and "1" of TextPairClassifier in Ranker

Patch to review:
diff --git a/docs/_src/usage/usage/ranker.md b/docs/_src/usage/usage/ranker.md
index d21dc21eea..e829dcfbfd 100644
--- a/docs/_src/usage/usage/ranker.md
+++ b/docs/_src/usage/usage/ranker.md
@@ -30,6 +30,7 @@ Alternatively, [this example](https://github.com/deepset-ai/FARM/blob/master/exa
 ### Description
 
 The FARMRanker consists of a Transformer-based model for document re-ranking using the TextPairClassifier of [FARM](https://github.com/deepset-ai/FARM).
+Given a text pair of query and passage, the TextPairClassifier either predicts label "1" if the pair is similar or label "0" if they are dissimilar (accompanied with a probability).
 While the underlying model can vary (BERT, Roberta, DistilBERT, ...), the interface remains the same.
 With a FARMRanker, you can:
 * Directly get predictions (re-ranked version of the supplied list of Document) via predict() if supplying a pre-trained model
diff --git a/docs/_src/usage/usage/reader.md b/docs/_src/usage/usage/reader.md
index f764805ffb..e1bbbc677a 100644
--- a/docs/_src/usage/usage/reader.md
+++ b/docs/_src/usage/usage/reader.md
@@ -247,7 +247,7 @@ When printing the full results of a Reader,
 you will see that each prediction is accompanied 
 by a value in the range of 0 to 1 reflecting the model's confidence in that prediction
 
-In the output of `print_answers()`, you will find the model confidence in dictionary key called `probability`.
+In the output of `print_answers()`, you will find the model confidence in dictionary key called `confidence`.
 
 ```python
 from haystack.utils import print_answers
@@ -263,7 +263,7 @@ print_answers(prediction, details="all")
                        'She travels with her father, Eddard, to '
                        "King's Landing when he is made Hand of the "
                        'King. Before she leaves,',
-            'probability': 0.9899835586547852,
+            'confidence': 0.9899835586547852,
             ...
         },
     ]
@@ -271,9 +271,14 @@ print_answers(prediction, details="all")
 ```
 
 In order to align this probability score with the model's accuracy, finetuning needs to be performed
-on a specific dataset. Have a look at this [FARM tutorial](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering_confidence.py)
-to see how this is done. 
-Note that a finetuned confidence score is specific to the domain that its finetuned on. 
+on a specific dataset. 
+To this end, the reader has a `calibrate_confidence_scores()` method. 
+This method needs the has the same input parameters as the `eval()` method because the calibration of confidence scores is performed on a dataset that comes with gold labels.
+The calibration therefore needs a DocumentStore containing labeled questions and evaluation documents.
+
+Have a look at this [FARM tutorial](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering_confidence.py)
+to see how to compare calibrated confidence scores with uncalibrated confidence scores within FARM. 
+Note that a finetuned confidence score is specific to the domain that it is finetuned on. 
 There is no guarantee that this performance can transfer to a new domain.
 
 Having a confidence score is particularly useful in cases where you need Haystack to work with a certain accuracy threshold.
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
index e043296889..0db1d8ea0a 100644
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -449,6 +449,7 @@ def eval(
             label_index: str = "label",
             doc_index: str = "eval_document",
             label_origin: str = "gold_label",
+            calibrate_conf_scores: bool = False
     ):
         """
         Performs evaluation on evaluation documents in the DocumentStore.
@@ -461,6 +462,8 @@ def eval(
         :param device: The device on which the tensors should be processed. Choose from "cpu" and "cuda".
         :param label_index: Index/Table name where labeled questions are stored
         :param doc_index: Index/Table name where documents that are used for evaluation are stored
+        :param label_origin: Field name where the gold labels are stored
+        :param calibrate_conf_scores: Whether to calibrate the temperature for temperature scaling of the confidence scores
         """
 
         if self.top_k_per_candidate != 4:
@@ -543,7 +546,7 @@ def eval(
 
         evaluator = Evaluator(data_loader=data_loader, tasks=self.inferencer.processor.tasks, device=device)
 
-        eval_results = evaluator.eval(self.inferencer.model)
+        eval_results = evaluator.eval(self.inferencer.model, calibrate_conf_scores=calibrate_conf_scores)
         toc = perf_counter()
         reader_time = toc - tic
         results = {
@@ -603,6 +606,30 @@ def _extract_answers_of_predictions(self, predictions: List[QAPred], top_k: Opti
 
         return answers, max_no_ans_gap
 
+    def calibrate_confidence_scores(
+            self,
+            document_store: BaseDocumentStore,
+            device: str,
+            label_index: str = "label",
+            doc_index: str = "eval_document",
+            label_origin: str = "gold_label"
+    ):
+        """
+        Calibrates confidence scores on evaluation documents in the DocumentStore.
+
+        :param document_store: DocumentStore containing the evaluation documents
+        :param device: The device on which the tensors should be processed. Choose from "cpu" and "cuda".
+        :param label_index: Index/Table name where labeled questions are stored
+        :param doc_index: Index/Table name where documents that are used for evaluation are stored
+        :param label_origin: Field name where the gold labels are stored
+        """
+        self.eval(document_store=document_store,
+                  device=device,
+                  label_index=label_index,
+                  doc_index=doc_index,
+                  label_origin=label_origin,
+                  calibrate_conf_scores=True)
+
     @staticmethod
     def _get_pseudo_prob(score: float):
         return float(expit(np.asarray(score) / 8))


Code review comments:
diff_hunk: @@ -263,17 +263,22 @@ print_answers(prediction, details="all")
                        'She travels with her father, Eddard, to '
                        "King's Landing when he is made Hand of the "
                        'King. Before she leaves,',
-            'probability': 0.9899835586547852,
+            'confidence': 0.9899835586547852,
             ...
         },
     ]
 }
 ```
 
 In order to align this probability score with the model's accuracy, finetuning needs to be performed
-on a specific dataset. Have a look at this [FARM tutorial](https://github.com/deepset-ai/FARM/blob/master/examples/question_answering_confidence.py)
-to see how this is done. 
-Note that a finetuned confidence score is specific to the domain that its finetuned on. 
+on a specific dataset. 
+To this end, the reader has a `calibrate_confidence_scores()` method. 
+This method needs the has the same input parameters as the `eval()` method because the calibration of confidence scores is performed on a dataset that comes with gold labels.
path: docs/_src/usage/usage/reader.md
text: @user1:
Typo: "needs the has the.."

"has the same input parameters as the `eval()` method" => Let's avoid that users start here to search for the eval method to understand what params this one expects. Let's rather put all information needed directly here or linking the API reference for calibrate_confidence_scores(). I guess you are already trying to say that documentstore is the only major parameter needed here, but I wouldn't really understand it that way by just reading the text.