Repository: deepset-ai/haystack

Issue:
Simplify usage of confidence scores & add documentation
With # we introduced an option to get more reliable confidence scores for predictions from a QA model / reader. 

So far the usage in Haystack seems not well documented and calibrating scores is not easily accessible. 
Calibration would probably look something like this: 

```
    evaluator_dev = Evaluator(
        data_loader=data_silo.get_data_loader("dev"),
        tasks=data_silo.processor.tasks,
        device=device
    )
    result_dev = evaluator_dev.eval(model, return_preds_and_labels=True, calibrate_conf_scores=True)

```

Or you set it manually to:
```
model.prediction_heads[0].temperature_for_confidence
```

Proposed changes:
1. Let's simplify the calibration by introducing a new "high-level" method in FARMReader (e.g. `calibrate_confidence_scores()`)
2. Let's update documentation so that more users are aware of this functionality (doc strings, but also adding an example either somewhere in the "usage section" or in one of the existing tutorials)



Discussion:

Resolving patch:
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -449,6 +449,7 @@ def eval(
             label_index: str = "label",
             doc_index: str = "eval_document",
             label_origin: str = "gold_label",
+            calibrate_conf_scores: bool = False
     ):
         """
         Performs evaluation on evaluation documents in the DocumentStore.
@@ -461,6 +462,8 @@ def eval(
         :param device: The device on which the tensors should be processed. Choose from "cpu" and "cuda".
         :param label_index: Index/Table name where labeled questions are stored
         :param doc_index: Index/Table name where documents that are used for evaluation are stored
+        :param label_origin: Field name where the gold labels are stored
+        :param calibrate_conf_scores: Whether to calibrate the temperature for temperature scaling of the confidence scores
         """
 
         if self.top_k_per_candidate != 4:
@@ -543,7 +546,7 @@ def eval(
 
         evaluator = Evaluator(data_loader=data_loader, tasks=self.inferencer.processor.tasks, device=device)
 
-        eval_results = evaluator.eval(self.inferencer.model)
+        eval_results = evaluator.eval(self.inferencer.model, calibrate_conf_scores=calibrate_conf_scores)
         toc = perf_counter()
         reader_time = toc - tic
         results = {
@@ -603,6 +606,30 @@ def _extract_answers_of_predictions(self, predictions: List[QAPred], top_k: Opti
 
         return answers, max_no_ans_gap
 
+    def calibrate_confidence_scores(
+            self,
+            document_store: BaseDocumentStore,
+            device: str,
+            label_index: str = "label",
+            doc_index: str = "eval_document",
+            label_origin: str = "gold_label"
+    ):
+        """
+        Calibrates confidence scores on evaluation documents in the DocumentStore.
+
+        :param document_store: DocumentStore containing the evaluation documents
+        :param device: The device on which the tensors should be processed. Choose from "cpu" and "cuda".
+        :param label_index: Index/Table name where labeled questions are stored
+        :param doc_index: Index/Table name where documents that are used for evaluation are stored
+        :param label_origin: Field name where the gold labels are stored
+        """
+        self.eval(document_store=document_store,
+                  device=device,
+                  label_index=label_index,
+                  doc_index=doc_index,
+                  label_origin=label_origin,
+                  calibrate_conf_scores=True)
+
     @staticmethod
     def _get_pseudo_prob(score: float):
         return float(expit(np.asarray(score) / 8))


Test patch: