Repository: deepset-ai/haystack
Pull request: Make returning predictions in evaluation possible
This PR adds an option for Finder and Retriever evaluation to return the predictions when doing evaluation.

Problem statement:
Returning the evaluation result as a dictionary
I would like to propose a feature for **evaluation of finder**, currently, in the finder evaluation, we receive summary results comprising of evaluation metrics and answers missed out by retriever and reader. I would like to know whether it is possible to get evaluation for question stored in form of a dictionary similar to finder.get_answer (here it is done for a question and it returns list answers with score and probability) , for example if one does evaluation for 250 questions, with this feature, the finder evaluation will return a dictionary of 250 lists of returned answers for their respective question.

Hints from issue discussion:
Should be possible. What would be your use case? Calculating own metrics on the results?
Hello @tholor . I have **two** use case in particular. First I want to check **Exact Match** of my answers, there could be a case, the returned answer **skips** last portion of the string (as given in my evaluation json file), this I am denoting as **case 1** , and **case 2** will be the predicted answer **collects** few more words instead of single word. In my project, **case 1** is a big issue, I would like to see **number of cases** falling in **category case 1**.

Second;y, I want to check **ranking** of my desired answer, if my desired answer is present in **top 3 answers** , I can keep **top-k-reader** = 3 .

So these are use cases which will give more information on the evaluation metrics. 
Ok got it. I see the value for this and I think we should add an option to return this data for detailed inspection of results and model "debugging". The default should still be the current behaviour in order to keep memory consumption low in production settings / large eval datasets.
  
Do you want to start a PR yourself or shall we work on it on our side? 
Hello @tholor , thank you for your response and I do concur with your idea of debugging mode. Also, I prefer not to start PR by myself, I will wait for implementation from your team.
Ok sure, we'll take care of it.

Commit message: Make returning preds in evaluation possible

Patch to review:
diff --git a/haystack/finder.py b/haystack/finder.py
index 4ead6911c7..c14ef75088 100644
--- a/haystack/finder.py
+++ b/haystack/finder.py
@@ -121,6 +121,7 @@ def eval(
         label_origin: str = "gold_label",
         top_k_retriever: int = 10,
         top_k_reader: int = 10,
+        return_preds: bool = False,
     ):
         """
         Evaluation of the whole pipeline by first evaluating the Retriever and then evaluating the Reader on the result
@@ -165,6 +166,9 @@ def eval(
         :type top_k_retriever: int
         :param top_k_reader: How many answers to return per question
         :type top_k_reader: int
+        :param return_preds: Whether to add predictions in the returned dictionary. If True, the returned dictionary
+                             contains the keys "predictions" and "metrics".
+        :type return_preds: bool
         """
 
         if not self.reader or not self.retriever:
@@ -205,6 +209,7 @@ def eval(
         previous_return_no_answers = self.reader.return_no_answers
         self.reader.return_no_answers = True
 
+        predictions = []
         # extract answers
         reader_start_time = time.time()
         for q_idx, question_docs in enumerate(questions_with_docs):
@@ -215,8 +220,10 @@ def eval(
             question_string = question.question
             docs = question_docs["docs"]  # type: ignore
             single_reader_start = time.time()
-            predicted_answers = self.reader.predict(question_string, docs, top_k=top_k_reader)  # type: ignore
+            predicted_answers = self.reader.predict(question_string, docs, top_k=top_k_reader) # type: ignore
             read_times.append(time.time() - single_reader_start)
+            if return_preds:
+                predictions.append(predicted_answers)
             counts = eval_counts_reader(question, predicted_answers, counts)
 
         counts["number_of_has_answer"] = counts["correct_retrievals"] - counts["number_of_no_answer"]
@@ -240,7 +247,10 @@ def eval(
         eval_results["avg_reader_time"] = mean(read_times)
         eval_results["total_finder_time"] = finder_total_time
 
-        return eval_results
+        if return_preds:
+            return {"metrics": eval_results, "predictions": predictions}
+        else:
+            return eval_results
 
     def eval_batch(
         self,
@@ -249,7 +259,8 @@ def eval_batch(
         label_origin: str = "gold_label",
         top_k_retriever: int = 10,
         top_k_reader: int = 10,
-        batch_size: int = 50
+        batch_size: int = 50,
+        return_preds: bool = False,
     ):
         """
         Evaluation of the whole pipeline by first evaluating the Retriever and then evaluating the Reader on the result
@@ -296,10 +307,13 @@ def eval_batch(
         :type top_k_reader: int
         :param batch_size: Number of samples per batch computed at once
         :type batch_size: int
+        :param return_preds: Whether to add predictions in the returned dictionary. If True, the returned dictionary
+                             contains the keys "predictions" and "metrics".
+        :type return_preds: bool
         """
 
         if not self.reader or not self.retriever:
-            raise Exception("Finder needs to have a reader and retriever for the evalutaion.")
+            raise Exception("Finder needs to have a reader and retriever for the evaluation.")
 
         counts = defaultdict(float)  # type: Dict[str, float]
         finder_start_time = time.time()
@@ -344,7 +358,10 @@ def eval_batch(
         logger.info(f"{number_of_questions - correct_retrievals} questions could not be answered due to the retriever.")
         logger.info(f"{correct_retrievals - counts['correct_readings_topk']} questions could not be answered due to the reader.")
 
-        return results
+        if return_preds:
+            return {"metrics": results, "predictions": predictions}
+        else:
+            return results
 
 
     def _retrieve_docs(self, questions: List[MultiLabel], top_k: int, doc_index: str):
@@ -363,7 +380,10 @@ def _retrieve_docs(self, questions: List[MultiLabel], top_k: int, doc_index: str
 
 
     @staticmethod
-    def print_eval_results(finder_eval_results: Dict):
+    def print_eval_results(finder_eval_results: Dict, contains_preds: bool = False):
+        if contains_preds:
+            finder_eval_results = finder_eval_results["metrics"]
+
         print("\n___Retriever Metrics in Finder___")
         print(f"Retriever Recall            : {finder_eval_results['retriever_recall']:.3f}")
         print(f"Retriever Mean Avg Precision: {finder_eval_results['retriever_map']:.3f}")
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
index 36047460bb..6c2bf7e902 100644
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -291,7 +291,7 @@ def predict_batch(self, question_doc_list: List[dict], top_k_per_question: int =
         result = []
         for idx, group in enumerate(grouped_predictions):
             answers, max_no_ans_gap = self._extract_answers_of_predictions(group, top_k_per_question)
-            question = group[0]
+            question = group[0].question
             cur_label = labels[idx]
             result.append({
                 "question": question,
diff --git a/haystack/retriever/base.py b/haystack/retriever/base.py
index e73492f8f8..21e97b4493 100644
--- a/haystack/retriever/base.py
+++ b/haystack/retriever/base.py
@@ -45,7 +45,8 @@ def eval(
         doc_index: str = "eval_document",
         label_origin: str = "gold_label",
         top_k: int = 10,
-        open_domain: bool = False
+        open_domain: bool = False,
+        return_preds: bool = False,
     ) -> dict:
         """
         Performs evaluation on the Retriever.
@@ -65,6 +66,8 @@ def eval(
                             contained in the retrieved docs (common approach in open-domain QA).
                             If ``False``, retrieval uses a stricter evaluation that checks if the retrieved document ids
                             are within ids explicitly stated in the labels.
+        :param return_preds: Whether to add predictions in the returned dictionary. If True, the returned dictionary
+                             contains the keys "predictions" and "metrics".
         """
 
         # Extract all questions for evaluation
@@ -86,11 +89,15 @@ def eval(
                 deduplicated_doc_ids = list(set([str(x) for x in label.multiple_document_ids]))
                 question_label_dict[label.question] = deduplicated_doc_ids
 
+        predictions = []
+
         # Option 1: Open-domain evaluation by checking if the answer string is in the retrieved docs
         logger.info("Performing eval queries...")
         if open_domain:
             for question, gold_answers in tqdm(question_label_dict.items()):
                 retrieved_docs = timed_retrieve(question, top_k=top_k, index=doc_index)
+                if return_preds:
+                    predictions.append({"question": question, "retrieved_docs": retrieved_docs})
                 # check if correct doc in retrieved docs
                 for doc_idx, doc in enumerate(retrieved_docs):
                     for gold_answer in gold_answers:
@@ -102,6 +109,8 @@ def eval(
         else:
             for question, gold_ids in tqdm(question_label_dict.items()):
                 retrieved_docs = timed_retrieve(question, top_k=top_k, index=doc_index)
+                if return_preds:
+                    predictions.append({"question": question, "retrieved_docs": retrieved_docs})
                 # check if correct doc in retrieved docs
                 for doc_idx, doc in enumerate(retrieved_docs):
                     for gold_id in gold_ids:
@@ -117,4 +126,15 @@ def eval(
         logger.info((f"For {correct_retrievals} out of {number_of_questions} questions ({recall:.2%}), the answer was in"
                      f" the top-{top_k} candidate passages selected by the retriever."))
 
-        return {"recall": recall, "map": mean_avg_precision, "retrieve_time": self.retrieve_time, "n_questions": number_of_questions, "top_k": top_k}
\ No newline at end of file
+        metrics =  {
+            "recall": recall,
+            "map": mean_avg_precision,
+            "retrieve_time": self.retrieve_time,
+            "n_questions": number_of_questions,
+            "top_k": top_k
+        }
+
+        if return_preds:
+            return {"metrics": metrics, "predictions": predictions}
+        else:
+            return metrics
\ No newline at end of file


Code review comments:
diff_hunk: @@ -363,7 +380,10 @@ def _retrieve_docs(self, questions: List[MultiLabel], top_k: int, doc_index: str
 
 
     @staticmethod
-    def print_eval_results(finder_eval_results: Dict):
+    def print_eval_results(finder_eval_results: Dict, contains_preds: bool = False):
+        if contains_preds:
path: haystack/finder.py
text: @user1:
let's rather do an automated check here if finder_eval_results.keys() contains  "predictions"