Repository: deepset-ai/haystack
Pull request: Provide Evaluation

Problem statement:
Adding evaluation to Reader, Retriever & Finder 
In order to compare different QA pipelines, evaluation is a core requirement.  
While this is quite simple on the isolated Reader, we are more interested in the big picture of the whole pipeline & the interaction with the retriever. 

## Requirements: 
- Eval metrics of retriever should reflect production usage
- It would be nice to use Squad style eval datasets (e.g. from the annotation tool or existing datasets) 

## Challenges: 
- We need indexing of docs at some point because we want to eval also objects like ElasticsearchRetriever
- Comparison of metrics between runs (e.g. new data was added)
- We don't want to impact the search experience in production (e.g. showing pure eval docs in search results)


# Sketched Approach:
- Indexing of eval data in DocumentStore in two indices (one for docs, one for annotations)
- Subsequent eval on those 

## DocumentStore.add_eval_data(data=Squad-like-file, index=”eval_document”, embedding_retriever=None)
Required before any eval
1. Adds documents from squad to index (default: new “eval_document”, also possible: existing “document”)
- Reason for separate eval index: No duplicates, Stable metrics, Not adding pure eval documents for “production search”
- Use additional "meta-data" fields if available in squad file, e.g.
"paragraphs": [
        {
          "context": "text",
          **"id": "abc"
          "company": "Texas"
          "year": 2017
          "filename": "some.pdf"**
          "qas": [...]
2. Adds annotations belonging to these docs to our existing “feedback” index
- Add field “index_name” to each entry (“foreign key”)
- Add field “origin” to differentiate between these eval labels and live user feedback (e.g. “gold_label” vs.”user_feedback”)

(3. Optional): copy some docs from production index “documents” as negative samples 

## Retriever.eval(label_index=”feedback”, doc_index=”eval_document”, label_origin=”gold_label” ... )
- Retriever.eval works on the passed index (i.e. index = “feedback”)
- Iterates over all questions in that index
- Calls retrieve() and checks whether we retrieve the right passages via comparison of doc_ids
- Metrics:
-- Recall ("top-n-retrieval accuracy")
-- Print non-ML info: For X out of X questions (x %), the answer was in the candidate passages selected by the retriever" 
-- (MAP)

## Reader.eval_on_file()
- Top_n recall, F1, EM
=> Re-use FARM functions for FARMReader?
=> Input: kind of eval dataset. SQuAD format?
(=> Transformer functions for TransformerReader?) 
(Later extra: Eval with Cross validation of reader)

## Reader.eval(index = “feedback”)
- Iterates over questions in index
- Get document text for question via document_id
- Calls reader with question + document text
- Evaluates metrics

## Finder.eval()
- Combination of Retriever+ Reader eval
- Show metrics on whole pipeline

Hints from issue discussion:

Commit message: Add data for tutorial to S3

Patch to review:
diff --git a/README.rst b/README.rst
index 9cc77688f2..3d1167b25c 100644
--- a/README.rst
+++ b/README.rst
@@ -55,10 +55,13 @@ Components
 
 Resources
 =========
-- Tutorial 1  - Basic QA Pipeline: `Jupyter notebook  <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial1_Basic_QA_Pipeline.ipynb>`__  or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial1_Basic_QA_Pipeline.ipynb>`__
-- Tutorial 2  - Fine-tuning a model on own data: `Jupyter notebook <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial2_Finetune_a_model_on_your_data.ipynb>`__ or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial2_Finetune_a_model_on_your_data.ipynb>`__
-- Tutorial 3  - Basic QA Pipeline without Elasticsearch: `Jupyter notebook <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial3_Basic_QA_Pipeline_without_Elasticsearch.ipynb>`__ or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial3_Basic_QA_Pipeline_without_Elasticsearch.ipynb>`__
+
+- Tutorial 1  - Basic QA Pipeline: `Jupyter notebook  <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial1_Basic_QA_Pipeline.ipynb>`_  or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial1_Basic_QA_Pipeline.ipynb>`_
+- Tutorial 2  - Fine-tuning a model on own data: `Jupyter notebook <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial2_Finetune_a_model_on_your_data.ipynb>`_ or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial2_Finetune_a_model_on_your_data.ipynb>`_
+- Tutorial 3  - Basic QA Pipeline without Elasticsearch: `Jupyter notebook <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial3_Basic_QA_Pipeline_without_Elasticsearch.py>`_ or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/update-tutorials/tutorials/Tutorial3_Basic_QA_Pipeline_without_Elasticsearch.ipynb>`_
 - Tutorial 4  - FAQ-style QA: `Jupyter notebook <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial4_FAQ_style_QA.ipynb>`__ or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial4_FAQ_style_QA.ipynb>`__
+- Tutorial 5  - Evaluation of the whole QA-Pipeline: `Jupyter noteboook <https://github.com/deepset-ai/haystack/blob/master/tutorials/Tutorial5_Evaluation.ipynb>`_ or `Colab <https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial5_Evaluation.ipynb>`_
+
 
 Quick Start
 ===========
diff --git a/haystack/database/elasticsearch.py b/haystack/database/elasticsearch.py
index 650d2b30bc..479bf191a0 100644
--- a/haystack/database/elasticsearch.py
+++ b/haystack/database/elasticsearch.py
@@ -108,8 +108,12 @@ def query(
         filters: dict = None,
         top_k: int = 10,
         custom_query: str = None,
+        index: str = None,
     ) -> [Document]:
 
+        if index is None:
+            index = self.index
+
         if custom_query:  # substitute placeholder for question and filters for the custom_query template string
             template = Template(custom_query)
 
@@ -145,7 +149,7 @@ def query(
             body["_source"] = {"excludes": self.excluded_meta_data}
 
         logger.debug(f"Retriever query: {body}")
-        result = self.client.search(index=self.index, body=body)["hits"]["hits"]
+        result = self.client.search(index=index, body=body)["hits"]["hits"]
 
         documents = [self._convert_es_hit_to_document(hit) for hit in result]
         return documents
@@ -199,3 +203,76 @@ def _convert_es_hit_to_document(self, hit, score_adjustment=0) -> Document:
             query_score=hit["_score"] + score_adjustment if hit["_score"] else None,
         )
         return document
+
+    def add_eval_data(self, filename: str, doc_index: str = "eval_document", label_index: str = "feedback"):
+        """
+        Adds a SQuAD-formatted file to the DocumentStore in order to be able to perform evaluation on it.
+
+        :param filename: Name of the file containing evaluation data
+        :type filename: str
+        :param doc_index: Elasticsearch index where evaluation documents should be stored
+        :type doc_index: str
+        :param label_index: Elasticsearch index where labeled questions should be stored
+        :type label_index: str
+        """
+
+        eval_docs_to_index = []
+        questions_to_index = []
+
+        with open(filename, "r") as file:
+            data = json.load(file)
+            for document in data["data"]:
+                for paragraph in document["paragraphs"]:
+                    doc_to_index= {}
+                    id = hash(paragraph["context"])
+                    for fieldname, value in paragraph.items():
+                        # write docs to doc_index
+                        if fieldname == "context":
+                            doc_to_index[self.text_field] = value
+                            doc_to_index["doc_id"] = str(id)
+                            doc_to_index["_op_type"] = "create"
+                            doc_to_index["_index"] = doc_index
+                        # write questions to label_index
+                        elif fieldname == "qas":
+                            for qa in value:
+                                question_to_index = {
+                                    "question": qa["question"],
+                                    "answers": qa["answers"],
+                                    "doc_id": str(id),
+                                    "origin": "gold_label",
+                                    "index_name": doc_index,
+                                    "_op_type": "create",
+                                    "_index": label_index
+                                }
+                                questions_to_index.append(question_to_index)
+                        # additional fields for docs
+                        else:
+                            doc_to_index[fieldname] = value
+
+                    for key, value in document.items():
+                        if key == "title":
+                            doc_to_index[self.name_field] = value
+                        elif key != "paragraphs":
+                            doc_to_index[key] = value
+
+                    eval_docs_to_index.append(doc_to_index)
+
+        bulk(self.client, eval_docs_to_index)
+        bulk(self.client, questions_to_index)
+
+    def get_all_docs_in_index(self, index, filters=None):
+        body = {
+            "query": {
+                "bool": {
+                    "must": {
+                        "match_all" : {}
+                    }
+                }
+            }
+        }
+
+        if filters:
+           body["query"]["bool"]["filter"] = {"term": filters}
+        result = scan(self.client, query=body, index=index)
+
+        return result
\ No newline at end of file
diff --git a/haystack/finder.py b/haystack/finder.py
index 9cf047c870..a0001726da 100644
--- a/haystack/finder.py
+++ b/haystack/finder.py
@@ -2,6 +2,8 @@
 
 import numpy as np
 from scipy.special import expit
+import time
+from statistics import mean
 
 logger = logging.getLogger(__name__)
 
@@ -96,3 +98,164 @@ def get_answers_via_similar_questions(self, question: str, top_k_retriever: int
             results["answers"].append(cur_answer)
 
         return results
+
+    def eval(self, label_index: str = "feedback", doc_index: str = "eval_document", label_origin: str = "gold_label",
+             top_k_retriever: int = 10, top_k_reader: int = 10):
+        """
+        Evaluation of the whole pipeline by first evaluating the Retriever and then evaluating the Reader on the result
+        of the Retriever.
+
+        Returns a dict containing the following metrics:
+            - "retriever_recall": Proportion of questions for which correct document is among retrieved documents
+            - "retriever_map": Mean of average precision for each question. Rewards retrievers that give relevant
+              documents a higher rank.
+            - "reader_recall": Proportion of predicted answers that overlap with correct answer
+            - "reader_map": Reader re-ranks documents retrieved by Retriever. Mean average precision of the re-ranking.
+            - "reader_em": Proportion of exact matches of predicted answers with their corresponding correct answers
+            - "reader_f1": Average overlap between predicted answers and their corresponding correct answers
+            - "total_retrieve_time": Time retriever needed to retrieve documents for all questions
+            - "avg_retrieve_time": Average time needed to retrieve documents for one question
+            - "total_reader_time": Time reader needed to extract answer out of retrieved documents for all questions
+                                   where the correct document is among the retrieved ones
+            - "avg_reader_time": Average time needed to extract answer out of retrieved documents for one question
+            - "total_finder_time": Total time for whole pipeline
+
+        :param label_index: Elasticsearch index where labeled questions are stored
+        :type label_index: str
+        :param doc_index: Elasticsearch index where documents that are used for evaluation are stored
+        :type doc_index: str
+        :param top_k_retriever: How many documents per question to return and pass to reader
+        :type top_k_retriever: int
+        :param top_k_reader: How many answers to return per question
+        :type top_k_reader: int
+        """
+        finder_start_time = time.time()
+        # extract all questions for evaluation
+        filter = {"origin": label_origin}
+        questions = self.retriever.document_store.get_all_docs_in_index(index=label_index, filters=filter)
+
+        correct_retrievals = 0
+        correct_readings = 0
+        summed_avg_precision_retriever = 0
+        summed_avg_precision_reader = 0
+        exact_matches = 0
+        summed_f1 = 0
+        retrieve_times = []
+        read_times = []
+
+        # retrieve documents
+        questions_with_docs = []
+        retriever_start_time = time.time()
+        for q_idx, question in enumerate(questions):
+            question_string = question["_source"]["question"]
+            single_retrieve_start = time.time()
+            retrieved_docs = self.retriever.retrieve(question_string, top_k=top_k_retriever, index=doc_index)
+            retrieve_times.append(time.time() - single_retrieve_start)
+            for doc_idx, doc in enumerate(retrieved_docs):
+                # check if correct doc among retrieved docs
+                if doc.meta["doc_id"] == question["_source"]["doc_id"]:
+                    correct_retrievals += 1
+                    summed_avg_precision_retriever += 1 / (doc_idx + 1)
+                    questions_with_docs.append({
+                        "question": question,
+                        "docs": retrieved_docs,
+                        "correct_es_doc_id": doc.id})
+                    break
+        retriever_total_time = time.time() - retriever_start_time
+        number_of_questions = q_idx + 1
+
+        previous_top_k_per_candidate = self.reader.top_k_per_candidate
+        previous_return_no_answers = self.reader.return_no_answers
+        self.reader.top_k_per_candidate = 1
+        self.reader.return_no_answers = True
+        # extract answers
+        reader_start_time = time.time()
+        for q_idx, question in enumerate(questions_with_docs):
+            question_string = question["question"]["_source"]["question"]
+            docs = question["docs"]
+            single_reader_start = time.time()
+            predicted_answers = self.reader.predict(question_string, docs, top_k_reader)
+            read_times.append(time.time() - single_reader_start)
+            # check if question is answerable
+            if question["question"]["_source"]["answers"]:
+                for answer_idx, answer in enumerate(predicted_answers["answers"]):
+                    found_answer = False
+                    found_em = False
+                    best_f1 = 0
+                    # check if correct document
+                    if answer["document_id"] == question["correct_es_doc_id"]:
+                        summed_avg_precision_reader += 1 / (answer_idx + 1)
+                        gold_spans = [(gold_answer["answer_start"], gold_answer["answer_start"] + len(gold_answer["text"]) + 1)
+                                      for gold_answer in question["question"]["_source"]["answers"]]
+                        predicted_span = (answer["offset_start_in_doc"], answer["offset_end_in_doc"])
+
+                        for gold_span in gold_spans:
+                            # check if overlap between gold answer and predicted answer
+                            if not found_answer:
+                                if (gold_span[0] <= predicted_span[1]) and (predicted_span[0] <= gold_span[1]):
+                                    correct_readings += 1
+                                    found_answer = True
+                            # check for exact match
+                            if not found_em:
+                                if (gold_span[0] == predicted_span[0]) and (gold_span[1] == predicted_span[1]):
+                                    exact_matches += 1
+                                    found_em = True
+                            # calculate f1
+                            pred_indices = list(range(predicted_span[0], predicted_span[1] + 1))
+                            gold_indices = list(range(gold_span[0], gold_span[1] + 1))
+                            n_overlap = len([x for x in pred_indices if x in gold_indices])
+                            if pred_indices and gold_indices and n_overlap:
+                                precision = n_overlap / len(pred_indices)
+                                recall = n_overlap / len(gold_indices)
+                                current_f1 = (2 * precision * recall) / (precision + recall)
+                                if current_f1 > best_f1:
+                                    best_f1 = current_f1
+                        summed_f1 += best_f1
+
+                    if found_answer and found_em:
+                        break
+            # question not answerable
+            else:
+                # As question is not answerable, it is not clear how to compute average precision for this question.
+                # For now, we decided to calculate average precision based on the rank of 'no answer'.
+                for answer_idx, answer in enumerate(predicted_answers["answers"]):
+                    # check if 'no answer'
+                    if answer["answer"] is None:
+                        correct_readings += 1
+                        summed_avg_precision_reader += 1 / (answer_idx + 1)
+                        exact_matches += 1
+                        summed_f1 += 1
+                        break
+        reader_total_time = time.time() - reader_start_time
+        finder_total_time = time.time() - finder_start_time
+
+        retriever_recall = correct_retrievals / number_of_questions
+        retriever_map = summed_avg_precision_retriever / number_of_questions
+        reader_recall = correct_readings / correct_retrievals
+        reader_map = summed_avg_precision_reader / correct_retrievals
+        reader_em = exact_matches / correct_retrievals
+        reader_f1 = summed_f1 / correct_retrievals
+
+        self.reader.top_k_per_candidate = previous_top_k_per_candidate
+        self.reader.return_no_answers = previous_return_no_answers
+
+        logger.info((f"{correct_readings} out of {number_of_questions} questions were correctly answered "
+                     f"({(correct_readings/number_of_questions):.2%})."))
+        logger.info(f"{number_of_questions-correct_retrievals} questions could not be answered due to the retriever.")
+        logger.info(f"{correct_retrievals-correct_readings} questions could not be answered due to the reader.")
+
+        results = {
+            "retriever_recall": retriever_recall,
+            "retriever_map": retriever_map,
+            "reader_recall": reader_recall,
+            "reader_map": reader_map,
+            "reader_em": reader_em,
+            "reader_f1" : reader_f1,
+            "total_retrieve_time": retriever_total_time,
+            "avg_retrieve_time": mean(retrieve_times),
+            "total_reader_time": reader_total_time,
+            "avg_reader_time": mean(read_times),
+            "total_finder_time": finder_total_time
+        }
+
+        return results
\ No newline at end of file
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
index eb7fc03308..c1afc3c54f 100644
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -4,13 +4,16 @@
 import numpy as np
 from farm.data_handler.data_silo import DataSilo
 from farm.data_handler.processor import SquadProcessor
+from farm.data_handler.dataloader import NamedDataLoader
 from farm.infer import Inferencer
 from farm.modeling.optimization import initialize_optimizer
 from farm.train import Trainer
+from farm.eval import Evaluator
 from farm.utils import set_all_seeds, initialize_device_settings
 from scipy.special import expit
 
 from haystack.database.base import Document
+from haystack.database.elasticsearch import ElasticsearchDocumentStore
 
 logger = logging.getLogger(__name__)
 
@@ -272,6 +275,116 @@ def predict(self, question: str, documents: [Document], top_k: int = None):
 
         return result
 
+    def eval_on_file(self, data_dir: str, test_filename: str, device: str):
+        """
+        Performs evaluation on a SQuAD-formatted file.
+
+        Returns a dict containing the following metrics:
+            - "EM": exact match score
+            - "f1": F1-Score
+            - "top_n_recall": Proportion of predicted answers that overlap with correct answer
+
+        :param data_dir: The directory in which the test set can be found
+        :type data_dir: Path or str
+        :param test_filename: The name of the file containing the test data in SQuAD format.
+        :type test_filename: str
+        :param device: The device on which the tensors should be processed. Choose from "cpu" and "cuda".
+        :type device: str
+        """
+        eval_processor = SquadProcessor(
+            tokenizer=self.inferencer.processor.tokenizer,
+            max_seq_len=self.inferencer.processor.max_seq_len,
+            label_list=self.inferencer.processor.tasks["question_answering"]["label_list"],
+            metric=self.inferencer.processor.tasks["question_answering"]["metric"],
+            train_filename=None,
+            dev_filename=None,
+            dev_split=0,
+            test_filename=test_filename,
+            data_dir=Path(data_dir),
+        )
+
+        data_silo = DataSilo(processor=eval_processor, batch_size=self.inferencer.batch_size, distributed=False)
+        data_loader = data_silo.get_data_loader("test")
+
+        evaluator = Evaluator(data_loader=data_loader, tasks=eval_processor.tasks, device=device)
+
+        eval_results = evaluator.eval(self.inferencer.model)
+        results = {
+            "EM": eval_results[0]["EM"],
+            "f1": eval_results[0]["f1"],
+            "top_n_recall": eval_results[0]["top_n_recall"]
+        }
+        return results
+
+    def eval(self, document_store: ElasticsearchDocumentStore, device: str, label_index: str = "feedback",
+             doc_index: str = "eval_document", label_origin: str = "gold_label"):
+        """
+        Performs evaluation on evaluation documents in Elasticsearch DocumentStore.
+
+        Returns a dict containing the following metrics:
+            - "EM": Proportion of exact matches of predicted answers with their corresponding correct answers
+            - "f1": Average overlap between predicted answers and their corresponding correct answers
+            - "top_n_recall": Proportion of predicted answers that overlap with correct answer
+
+        :param document_store: The ElasticsearchDocumentStore containing the evaluation documents
+        :type document_store: ElasticsearchDocumentStore
+        :param device: The device on which the tensors should be processed. Choose from "cpu" and "cuda".
+        :type device: str
+        :param label_index: Elasticsearch index where labeled questions are stored
+        :type label_index: str
+        :param doc_index: Elasticsearch index where documents that are used for evaluation are stored
+        :type doc_index: str
+        """
+
+        # extract all questions for evaluation
+        filter = {"origin": label_origin}
+        questions = document_store.get_all_docs_in_index(index=label_index, filters=filter)
+
+        # mapping from doc_id to questions
+        doc_questions_dict = {}
+        id = 0
+        for question in questions:
+            doc_id = question["_source"]["doc_id"]
+            if doc_id not in doc_questions_dict:
+                doc_questions_dict[doc_id] = [{
+                    "id": id,
+                    "question" : question["_source"]["question"],
+                    "answers" : question["_source"]["answers"],
+                    "is_impossible" : False if question["_source"]["answers"] else True
+                }]
+            else:
+                doc_questions_dict[doc_id].append({
+                    "id": id,
+                    "question" : question["_source"]["question"],
+                    "answers" : question["_source"]["answers"],
+                    "is_impossible" : False if question["_source"]["answers"] else True
+                })
+            id += 1
+
+        # extract eval documents and convert data back to SQuAD-like format
+        documents = document_store.get_all_docs_in_index(index=doc_index)
+        dicts = []
+        for document in documents:
+            doc_id = document["_source"]["doc_id"]
+            text = document["_source"]["text"]
+            questions = doc_questions_dict[doc_id]
+            dicts.append({"qas" : questions, "context" : text})
+
+        # Create DataLoader that can be passed to the Evaluator
+        indices = range(len(dicts))
+        dataset, tensor_names = self.inferencer.processor.dataset_from_dicts(dicts, indices=indices)
+        data_loader = NamedDataLoader(dataset=dataset, batch_size=self.inferencer.batch_size, tensor_names=tensor_names)
+
+        evaluator = Evaluator(data_loader=data_loader, tasks=self.inferencer.processor.tasks, device=device)
+
+        eval_results = evaluator.eval(self.inferencer.model)
+        results = {
+            "EM": eval_results[0]["EM"],
+            "f1": eval_results[0]["f1"],
+            "top_n_recall": eval_results[0]["top_n_recall"]
+        }
+        return results
+
     @staticmethod
     def _calc_no_answer(no_ans_gaps,best_score_answer):
         # "no answer" scores and positive answers scores are difficult to compare, because
diff --git a/haystack/retriever/elasticsearch.py b/haystack/retriever/elasticsearch.py
index 37020fbe19..20c01f0878 100644
--- a/haystack/retriever/elasticsearch.py
+++ b/haystack/retriever/elasticsearch.py
@@ -41,12 +41,58 @@ def __init__(self, document_store: Type[BaseDocumentStore], custom_query: str =
         self.document_store = document_store
         self.custom_query = custom_query
 
-    def retrieve(self, query: str, filters: dict = None, top_k: int = 10) -> [Document]:
-        documents = self.document_store.query(query, filters, top_k, self.custom_query)
+    def retrieve(self, query: str, filters: dict = None, top_k: int = 10, index: str = None) -> [Document]:
+        if index is None:
+            index = self.document_store.index
+
+        documents = self.document_store.query(query, filters, top_k, self.custom_query, index)
         logger.info(f"Got {len(documents)} candidates from retriever")
 
         return documents
 
+    def eval(self, label_index: str = "feedback", doc_index: str = "eval_document", label_origin: str = "gold_label",
+             top_k: int = 10) -> dict:
+        """
+        Performs evaluation on the Retriever.
+        Retriever is evaluated based on whether it finds the correct document given the question string and at which
+        position in the ranking of documents the correct document is.
+
+        Returns a dict containing the following metrics:
+            - "recall": Proportion of questions for which correct document is among retrieved documents
+            - "mean avg precision": Mean of average precision for each question. Rewards retrievers that give relevant
+              documents a higher rank.
+
+        :param label_index: Elasticsearch index where labeled questions are stored
+        :param doc_index: Elasticsearch index where documents that are used for evaluation are stored
+        :param top_k: How many documents to return per question
+        """
+
+        # extract all questions for evaluation
+        filter = {"origin": label_origin}
+        questions = self.document_store.get_all_docs_in_index(index=label_index, filters=filter)
+
+        # calculate recall and mean-average-precision
+        correct_retrievals = 0
+        summed_avg_precision = 0
+        for q_idx, question in enumerate(questions):
+            question_string = question["_source"]["question"]
+            retrieved_docs = self.retrieve(question_string, top_k=top_k, index=doc_index)
+            # check if correct doc in retrieved docs
+            for doc_idx, doc in enumerate(retrieved_docs):
+                if doc.meta["doc_id"] == question["_source"]["doc_id"]:
+                    correct_retrievals += 1
+                    summed_avg_precision += 1 / (doc_idx + 1)
+                    break
+
+        number_of_questions = q_idx + 1
+        recall = correct_retrievals / number_of_questions
+        mean_avg_precision = summed_avg_precision / number_of_questions
+
+        logger.info((f"For {correct_retrievals} out of {number_of_questions} questions ({recall:.2%}), the answer was in"
+                     f" the top-{top_k} candidate passages selected by the retriever."))
+
+        return {"recall": recall, "mean avg precision": mean_avg_precision}
+
 
 class EmbeddingRetriever(BaseRetriever):
     def __init__(
diff --git a/tutorials/Tutorial5_Evaluation.ipynb b/tutorials/Tutorial5_Evaluation.ipynb
new file mode 100644
index 0000000000..e746315a2b
--- /dev/null
+++ b/tutorials/Tutorial5_Evaluation.ipynb
@@ -0,0 +1,1238 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Evalutaion\n",
+    "To be able to make a statement about the performance of a question-asnwering system, it is important to evalute it. Furthermore, evaluation allows to determine which parts of the system can be improved."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Start an Elasticsearch server\n",
+    "You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in your environment (eg., in Colab notebooks), then you can manually download and execute Elasticsearch from source."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "a844e3ec4f41b5d2b24fe3d562e8302896baea1d0a761295998434c2de490714\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Recommended: Start Elasticsearch using Docker\n",
+    "! docker run -d -p 9200:9200 -e \"discovery.type=single-node\" elasticsearch:7.6.2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# In Colab / No Docker environments: Start Elasticsearch from source\n",
+    "#! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-linux-x86_64.tar.gz -q\n",
+    "#! tar -xzf elasticsearch-7.6.2-linux-x86_64.tar.gz\n",
+    "#! chown -R daemon:daemon elasticsearch-7.6.2\n",
+    "\n",
+    "#import os\n",
+    "#from subprocess import Popen, PIPE, STDOUT\n",
+    "#es_server = Popen(['elasticsearch-7.6.2/bin/elasticsearch'],\n",
+    "                   stdout=PIPE, stderr=STDOUT,\n",
+    "                   preexec_fn=lambda: os.setuid(1)  # as daemon\n",
+    "                  )\n",
+    "# wait until ES has started\n",
+    "#! sleep 30"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:03:25 - INFO - farm.utils -   device: cuda n_gpu: 1, distributed training: False, automatic mixed precision training: None\n"
+     ]
+    }
+   ],
+   "source": [
+    "from farm.utils import initialize_device_settings\n",
+    "\n",
+    "device, n_gpu = initialize_device_settings(use_cuda=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "\n",
+    "from haystack.indexing.io import fetch_archive_from_http\n",
+    "\n",
+    "# Download evaluation data, which is a subset of Natural Questions development set containing 50 documents\n",
+    "doc_dir = \"../data/nq\"\n",
+    "s3_url = \"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/nq_dev_subset.json.zip\"\n",
+    "fetch_archive_from_http(url=s3_url, output_dir=doc_dir)"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Connect to Elasticsearch\n",
+    "from haystack.database.elasticsearch import ElasticsearchDocumentStore\n",
+    "\n",
+    "document_store = ElasticsearchDocumentStore(host=\"localhost\", username=\"\", password=\"\", create_index=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:03:37 - INFO - elasticsearch -   POST http://localhost:9200/_bulk [status:200 request:0.796s]\n",
+      "05/19/2020 09:03:38 - INFO - elasticsearch -   POST http://localhost:9200/_bulk [status:200 request:0.222s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Add evaluation data to Elasticsearch database\n",
+    "document_store.add_eval_data(\"../data/natural_questions/dev_subset.json\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initialize components of QA-System"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize Retriever\n",
+    "from haystack.retriever.elasticsearch import ElasticsearchRetriever\n",
+    "\n",
+    "retriever = ElasticsearchRetriever(document_store=document_store)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:03:46 - INFO - farm.utils -   device: cuda n_gpu: 1, distributed training: False, automatic mixed precision training: None\n",
+      "05/19/2020 09:03:46 - INFO - farm.infer -   Could not find `deepset/roberta-base-squad2` locally. Try to download from model hub ...\n",
+      "05/19/2020 09:03:50 - WARNING - farm.modeling.language_model -   Could not automatically detect from language model name what language it is. \n",
+      "\t We guess it's an *ENGLISH* model ... \n",
+      "\t If not: Init the language model by supplying the 'language' param.\n",
+      "05/19/2020 09:03:56 - WARNING - farm.modeling.prediction_head -   Some unused parameters are passed to the QuestionAnsweringHead. Might not be a problem. Params: {\"loss_ignore_index\": -1}\n",
+      "05/19/2020 09:04:02 - INFO - farm.utils -   device: cuda n_gpu: 1, distributed training: False, automatic mixed precision training: None\n",
+      "05/19/2020 09:04:02 - INFO - farm.infer -   Got ya 7 parallel workers to do inference ...\n",
+      "05/19/2020 09:04:02 - INFO - farm.infer -    0    0    0    0    0    0    0 \n",
+      "05/19/2020 09:04:02 - INFO - farm.infer -   /w\\  /w\\  /w\\  /w\\  /w\\  /w\\  /w\\\n",
+      "05/19/2020 09:04:02 - INFO - farm.infer -   /'\\  / \\  /'\\  /'\\  / \\  / \\  /'\\\n",
+      "05/19/2020 09:04:02 - INFO - farm.infer -               \n"
+     ]
+    }
+   ],
+   "source": [
+    "# Initialize Reader\n",
+    "from haystack.reader.farm import FARMReader\n",
+    "\n",
+    "reader = FARMReader(\"deepset/roberta-base-squad2\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize Finder which sticks together Reader and Retriever\n",
+    "from haystack.finder import Finder\n",
+    "\n",
+    "finder = Finder(reader, retriever)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluation of Retriever"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/feedback/_search?scroll=5m&size=1000 [status:200 request:0.090s]\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.051s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.013s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.012s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.012s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.013s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.010s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.010s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.009s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.011s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.010s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.010s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.009s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.009s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:11 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.010s]\n",
+      "05/19/2020 09:04:11 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.009s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   GET http://localhost:9200/_search/scroll [status:200 request:0.011s]\n",
+      "05/19/2020 09:04:12 - INFO - elasticsearch -   DELETE http://localhost:9200/_search/scroll [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:12 - INFO - haystack.retriever.elasticsearch -   For 59 out of 59 questions (100.00%), the answer was in the top-10 candidate passages selected by the retriever.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Retriever Recall: 1.0\n",
+      "Retriever Mean Avg Precision: 0.984934086629002\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Evaluate Retriever on its own\n",
+    "retriever_eval_results = retriever.eval()\n",
+    "\n",
+    "## Retriever Recall is the proportion of questions for which the correct document containing the answer is\n",
+    "## among the correct documents\n",
+    "print(\"Retriever Recall:\", retriever_eval_results[\"recall\"])\n",
+    "## Retriever Mean Avg Precision rewards retrievers that give relevant documents a higher rank\n",
+    "print(\"Retriever Mean Avg Precision:\", retriever_eval_results[\"mean avg precision\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluation of Reader"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:04:22 - INFO - elasticsearch -   GET http://localhost:9200/feedback/_search?scroll=5m&size=1000 [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:22 - INFO - elasticsearch -   GET http://localhost:9200/_search/scroll [status:200 request:0.003s]\n",
+      "05/19/2020 09:04:22 - INFO - elasticsearch -   DELETE http://localhost:9200/_search/scroll [status:200 request:0.001s]\n",
+      "05/19/2020 09:04:22 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search?scroll=5m&size=1000 [status:200 request:0.014s]\n",
+      "05/19/2020 09:04:22 - INFO - elasticsearch -   GET http://localhost:9200/_search/scroll [status:200 request:0.002s]\n",
+      "05/19/2020 09:04:22 - INFO - elasticsearch -   DELETE http://localhost:9200/_search/scroll [status:200 request:0.002s]\n",
+      "Evaluating: 100%|██████████| 64/64 [00:14<00:00,  4.28it/s]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Reader Top-N-Recall: 0.5084745762711864\n",
+      "Reader Exact Match: 0.23728813559322035\n",
+      "Reader F1-Score: 0.23728813559322035\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Evaluate Reader on its own\n",
+    "reader_eval_results = reader.eval(document_store=document_store, device=device)\n",
+    "\n",
+    "# Evaluation of Reader can also be done directly on a SQuAD-formatted file \n",
+    "# without passing the data to Elasticsearch\n",
+    "#reader_eval_results = reader.eval_on_file(\"../data/natural_questions\", \"dev_subset.json\", device=device)\n",
+    "\n",
+    "## Reader Top-N-Recall is the proportion of predicted answers that overlap with their corresponding correct answer\n",
+    "print(\"Reader Top-N-Recall:\", reader_eval_results[\"top_n_recall\"])\n",
+    "## Reader Exact Match is the proportion of questions where the predicted answer is exactly the same as the correct answer\n",
+    "print(\"Reader Exact Match:\", reader_eval_results[\"EM\"])\n",
+    "## Reader F1-Score is the average overlap between the predicted answers and the correct answers\n",
+    "print(\"Reader F1-Score:\", reader_eval_results[\"f1\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluation of Finder"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/feedback/_search?scroll=5m&size=1000 [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:57 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:57 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.008s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.007s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.004s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.004s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.005s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/eval_document/_search [status:200 request:0.006s]\n",
+      "05/19/2020 09:04:58 - INFO - haystack.retriever.elasticsearch -   Got 10 candidates from retriever\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   GET http://localhost:9200/_search/scroll [status:200 request:0.003s]\n",
+      "05/19/2020 09:04:58 - INFO - elasticsearch -   DELETE http://localhost:9200/_search/scroll [status:200 request:0.001s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.40 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.91 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.53 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.48 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.19 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.77 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.51 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.69 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.33 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.97 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.78 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.52 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.90 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.82 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.47 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.26 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.54 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.29 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  2.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.22 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  3.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.56 Batches/s]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 19.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.89 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  2.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:01<00:00,  1.50 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.58 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  2.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.94 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  3.99 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.75 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.39 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.13 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.91 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.67 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.88 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.82 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  3.18 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  2.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.16 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.77 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  2.06 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  3.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.08 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.88 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.91 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  8.07 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.69 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  9.20 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.31 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.66 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.14 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.29 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.00 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.00 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.96 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.51 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.42 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.97 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.11 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.54 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.89 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  9.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.22 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.52 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.20 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  3.75 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.45 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.96 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.58 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.07 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.37 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.84 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.63 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 18.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.54 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.58 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.06 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.73 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 11.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.90 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.63 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.33 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.18 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.08 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.16 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.49 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.66 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.21 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.16 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.74 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.31 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.27 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.36 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.96 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.34 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.82 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.25 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.49 Batches/s]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.01 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 11.02 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.27 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.53 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.87 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.84 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.74 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.36 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.73 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.39 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.39 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.61 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:01<00:00,  1.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  9.27 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.63 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.02 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.35 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.29 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.94 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.11 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.52 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.06 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.74 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.03 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.49 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.53 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.97 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.96 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.26 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 62.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 11.94 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.22 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 24.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.13 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.90 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.03 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.63 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.45 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.61 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.13 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.66 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.90 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.69 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.01 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.11 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.00 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.01 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.45 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.99 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.91 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.73 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.33 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.96 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.70 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.78 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.78 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.39 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.42 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.00 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.03 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.48 Batches/s]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.67 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 10.14 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.77 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.58 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.76 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 29.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.61 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.24 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 27.26 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.66 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 10.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.42 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.08 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  9.75 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.73 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.32 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.07 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.06 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.50 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 11.30 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.33 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 26.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.75 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.02 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.44 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.26 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.77 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.21 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.89 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.69 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.14 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.32 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  3.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.02 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 23.44 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.37 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.01 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.34 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.02 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.48 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.65 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 58.54 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.75 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.78 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.18 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.82 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.51 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.42 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.67 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.99 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.07 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.89 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  3.22 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.67 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.63 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.73 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.18 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.61 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.97 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.13 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.03 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.74 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.41 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.00 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 26.84 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.11 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.00 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.73 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.87 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.94 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.00 Batches/s]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.15 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.08 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.66 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.15 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.22 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.77 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.24 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.99 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.80 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.21 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 10.65 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.03 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.39 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.16 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  9.06 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.37 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.27 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  9.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.76 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.82 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.94 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 10.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.06 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.01 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.53 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.95 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.78 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.69 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.17 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.49 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.74 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.74 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.54 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 17.25 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 26.22 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.35 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.02 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.76 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.07 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.24 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.76 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 19.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.01 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.19 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.37 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.08 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.71 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.11 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.66 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.16 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.53 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.54 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.58 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.76 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.36 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.03 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.01 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  6.69 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.98 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.48 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.51 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.24 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.42 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.85 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.49 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.42 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.20 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:02<00:00,  1.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.15 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  1.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:01<00:00,  1.48 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  1.64 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.78 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:01<00:00,  1.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.97 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.26 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.21 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.25 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.89 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.23 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.93 Batches/s]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.07 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.50 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.18 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.75 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.62 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.49 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.52 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.59 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 17.37 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.15 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.11 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.89 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.04 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.90 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.88 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.65 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.39 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.30 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.87 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.75 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.42 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.12 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.70 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.50 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 12.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.30 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.21 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.45 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.94 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.15 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.91 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  3.46 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  2.29 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.68 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:01<00:00,  2.58 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.16 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.97 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  7.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.71 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.35 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.86 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  4.26 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.53 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.51 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.37 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 24.83 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  3.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.57 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.05 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 13.77 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.43 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.48 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.87 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.40 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.10 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.66 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  5.13 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.79 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 2/2 [00:00<00:00,  7.65 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  6.07 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  8.36 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.24 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.56 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.92 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  1.84 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  5.00 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  6.91 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.99 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.99 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.60 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 14.55 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.67 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  9.03 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  4.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.34 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 11.72 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 16.78 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 3/3 [00:00<00:00,  3.81 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 7/7 [00:01<00:00,  4.28 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  4.20 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  5.09 Batches/s]\n",
+      "Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00,  2.27 Batches/s]\n",
+      "05/19/2020 09:09:54 - INFO - haystack.finder -   57 out of 59 questions were correctly answered (96.61%).\n",
+      "05/19/2020 09:09:54 - INFO - haystack.finder -   0 questions could not be answered due to the retriever.\n",
+      "05/19/2020 09:09:54 - INFO - haystack.finder -   2 questions could not be answered due to the reader.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Retriever Recall in Finder: 1.0\n",
+      "Retriever Mean Avg Precision in Finder: 0.984934086629002\n",
+      "Reader Recall in Finder: 0.9661016949152542\n",
+      "Reader Mean Avg Precision in Finder: 0.44187516814635447\n",
+      "Reader Exact Match in Finder: 0.9661016949152542\n",
+      "Reader F1-Score in Finder: 0.9661016949152542\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Evaluate combination of Reader and Retriever through Finder\n",
+    "finder_eval_results = finder.eval()\n",
+    "\n",
+    "print(\"Retriever Recall in Finder:\", finder_eval_results[\"retriever_recall\"])\n",
+    "print(\"Retriever Mean Avg Precision in Finder:\", finder_eval_results[\"retriever_map\"])\n",
+    "\n",
+    "# Reader is only evaluated with those questions, where the correct document is among the retrieved ones\n",
+    "print(\"Reader Recall in Finder:\", finder_eval_results[\"reader_recall\"])\n",
+    "print(\"Reader Mean Avg Precision in Finder:\", finder_eval_results[\"reader_map\"])\n",
+    "print(\"Reader Exact Match in Finder:\", finder_eval_results[\"reader_em\"])\n",
+    "print(\"Reader F1-Score in Finder:\", finder_eval_results[\"reader_f1\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "name": "haystack",
+   "language": "python",
+   "display_name": "haystack"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
\ No newline at end of file
diff --git a/tutorials/Tutorial5_Evaluation.py b/tutorials/Tutorial5_Evaluation.py
new file mode 100644
index 0000000000..605480e262
--- /dev/null
+++ b/tutorials/Tutorial5_Evaluation.py
@@ -0,0 +1,80 @@
+from haystack.database.elasticsearch import ElasticsearchDocumentStore
+from haystack.indexing.io import fetch_archive_from_http
+from haystack.retriever.elasticsearch import ElasticsearchRetriever
+from haystack.reader.farm import FARMReader
+from haystack.finder import Finder
+from farm.utils import initialize_device_settings
+
+import logging
+import subprocess
+import time
+
+LAUNCH_ELASTICSEARCH = False
+device, n_gpu = initialize_device_settings(use_cuda=True)
+
+# Start an Elasticsearch server
+# You can start Elasticsearch on your local machine instance using Docker. If Docker is not readily available in
+# your environment (eg., in Colab notebooks), then you can manually download and execute Elasticsearch from source.
+if LAUNCH_ELASTICSEARCH:
+    logging.info("Starting Elasticsearch ...")
+    status = subprocess.run(
+        ['docker run -d -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.6.2'], shell=True
+    )
+    if status.returncode:
+        raise Exception("Failed to launch Elasticsearch. If you want to connect to an existing Elasticsearch instance"
+                        "then set LAUNCH_ELASTICSEARCH in the script to False.")
+    time.sleep(30)
+
+# Download evaluation data, which is a subset of Natural Questions development set containing 50 documents
+doc_dir = "../data/nq"
+s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/nq_dev_subset.json.zip"
+fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
+
+# Connect to Elasticsearch
+document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document", create_index=False)
+# Add evaluation data to Elasticsearch database
+document_store.add_eval_data("../data/nq/nq_dev_subset.json")
+
+# Initialize Retriever
+retriever = ElasticsearchRetriever(document_store=document_store)
+
+# Initialize Reader
+reader = FARMReader("deepset/roberta-base-squad2")
+
+# Initialize Finder which sticks together Reader and Retriever
+finder = Finder(reader, retriever)
+
+# Evaluate Retriever on its own
+retriever_eval_results = retriever.eval()
+## Retriever Recall is the proportion of questions for which the correct document containing the answer is
+## among the correct documents
+print("Retriever Recall:", retriever_eval_results["recall"])
+## Retriever Mean Avg Precision rewards retrievers that give relevant documents a higher rank
+print("Retriever Mean Avg Precision:", retriever_eval_results["mean avg precision"])
+
+# Evaluate Reader on its own
+reader_start = time.time()
+reader_eval_results = reader.eval(document_store=document_store, device=device)
+reader_total = time.time() - reader_start
+# Evaluation of Reader can also be done directly on a SQuAD-formatted file without passing the data to Elasticsearch
+#reader_eval_results = reader.eval_on_file("../data/natural_questions", "dev_subset.json", device=device)
+
+## Reader Top-N-Recall is the proportion of predicted answers that overlap with their corresponding correct answer
+print("Reader Top-N-Recall:", reader_eval_results["top_n_recall"])
+## Reader Exact Match is the proportion of questions where the predicted answer is exactly the same as the correct answer
+print("Reader Exact Match:", reader_eval_results["EM"])
+## Reader F1-Score is the average overlap between the predicted answers and the correct answers
+print("Reader F1-Score:", reader_eval_results["f1"])
+
+
+# Evaluate combination of Reader and Retriever through Finder
+finder_eval_results = finder.eval()
+print("Retriever Recall in Finder:", finder_eval_results["retriever_recall"])
+print("Retriever Mean Avg Precision in Finder:", finder_eval_results["retriever_map"])
+# Reader is only evaluated with those questions, where the correct document is among the retrieved ones
+print("Reader Recall in Finder:", finder_eval_results["reader_recall"])
+print("Reader Mean Avg Precision in Finder:", finder_eval_results["reader_map"])
+print("Reader Exact Match in Finder:", finder_eval_results["reader_em"])
+print("Reader F1-Score in Finder:", finder_eval_results["reader_f1"])
+
+print(f"Finder time: {finder_eval_results['total_finder_time']}s")


Code review comments:
diff_hunk: @@ -41,12 +41,58 @@ def __init__(self, document_store: Type[BaseDocumentStore], custom_query: str =
         self.document_store = document_store
         self.custom_query = custom_query
 
-    def retrieve(self, query: str, filters: dict = None, top_k: int = 10) -> [Document]:
-        documents = self.document_store.query(query, filters, top_k, self.custom_query)
+    def retrieve(self, query: str, filters: dict = None, top_k: int = 10, index: str = None) -> [Document]:
+        if index is None:
+            index = self.document_store.index
+
+        documents = self.document_store.query(query, filters, top_k, self.custom_query, index)
         logger.info(f"Got {len(documents)} candidates from retriever")
 
         return documents
 
+    def eval(self, label_index: str = "feedback", doc_index: str = "eval_document", label_origin: str = "gold_label",
+             top_k: int = 10) -> dict:
+        """
+        Performs evaluation on the Retriever.
+        Retriever is evaluated based on whether it finds the correct document given the question string and at which
+        position in the ranking of documents the correct document is.
+
+        Returns a dict containing the following metrics:
+            - "recall": Proportion of questions for which correct document is among retrieved documents
+            - "mean avg precision": Mean of average precision for each question. Rewards retrievers that give relevant
+              documents a higher rank.
+
+        :param label_index: Elasticsearch index where labeled questions are stored
path: haystack/retriever/elasticsearch.py
text: @user1:
While Elasticsearch is currently our recommended DocumentStore we should phrase this a bit more generic (e.g. "Index/Table in DocumentStore where labeled questions are stored"). If I got it right supporting eval for another DocumentStore only requires implementations of the methods "add_eval_data()" and "get_all_docs_in_index()".

diff_hunk: @@ -199,3 +203,76 @@ def _convert_es_hit_to_document(self, hit, score_adjustment=0) -> Document:
             query_score=hit["_score"] + score_adjustment if hit["_score"] else None,
         )
         return document
+
+    def add_eval_data(self, filename: str, doc_index: str = "eval_document", label_index: str = "feedback"):
+        """
+        Adds a SQuAD-formatted file to the DocumentStore in order to be able to perform evaluation on it.
+
+        :param filename: Name of the file containing evaluation data
+        :type filename: str
+        :param doc_index: Elasticsearch index where evaluation documents should be stored
+        :type doc_index: str
+        :param label_index: Elasticsearch index where labeled questions should be stored
+        :type label_index: str
+        """
+
+        eval_docs_to_index = []
+        questions_to_index = []
+
+        with open(filename, "r") as file:
+            data = json.load(file)
+            for document in data["data"]:
+                for paragraph in document["paragraphs"]:
+                    doc_to_index= {}
+                    id = hash(paragraph["context"])
+                    for fieldname, value in paragraph.items():
+                        # write docs to doc_index
+                        if fieldname == "context":
+                            doc_to_index[self.text_field] = value
+                            doc_to_index["doc_id"] = str(id)
+                            doc_to_index["_op_type"] = "create"
+                            doc_to_index["_index"] = doc_index
+                        # write questions to label_index
+                        elif fieldname == "qas":
+                            for qa in value:
+                                question_to_index = {
+                                    "question": qa["question"],
+                                    "answers": qa["answers"],
+                                    "doc_id": str(id),
+                                    "origin": "gold_label",
+                                    "index_name": doc_index,
+                                    "_op_type": "create",
+                                    "_index": label_index
+                                }
+                                questions_to_index.append(question_to_index)
+                        # additional fields for docs
+                        else:
+                            doc_to_index[fieldname] = value
+
+                    for key, value in document.items():
+                        if key == "title":
+                            doc_to_index[self.name_field] = value
+                        elif key != "paragraphs":
+                            doc_to_index[key] = value
+
+                    eval_docs_to_index.append(doc_to_index)
+
+        bulk(self.client, eval_docs_to_index)
+        bulk(self.client, questions_to_index)
+
+    def get_all_docs_in_index(self, index, filters=None):
path: haystack/database/elasticsearch.py
text: @user1:
Can we make this fct private?
Can we rename it? Since it applies filters on the docs and e.g. in the finder just returns the questions. That doesnt really fit the name.

@user2:
Wouldn't make it private as we call it from retriever.eval().
As the functionality is overlapping with "get_all_documents()", we could a) extend this one and get rid of the new implementation or b) rename the new one to "get_all_documents_in_index()

diff_hunk: @@ -96,3 +98,164 @@ def get_answers_via_similar_questions(self, question: str, top_k_retriever: int
             results["answers"].append(cur_answer)
 
         return results
+
+    def eval(self, label_index: str = "feedback", doc_index: str = "eval_document", label_origin: str = "gold_label",
+             top_k_retriever: int = 10, top_k_reader: int = 10):
+        """
+        Evaluation of the whole pipeline by first evaluating the Retriever and then evaluating the Reader on the result
+        of the Retriever.
+
+        Returns a dict containing the following metrics:
+            - "retriever_recall": Proportion of questions for which correct document is among retrieved documents
+            - "retriever_map": Mean of average precision for each question. Rewards retrievers that give relevant
+              documents a higher rank.
+            - "reader_recall": Proportion of predicted answers that overlap with correct answer
+            - "reader_map": Reader re-ranks documents retrieved by Retriever. Mean average precision of the re-ranking.
+            - "reader_em": Proportion of exact matches of predicted answers with their corresponding correct answers
+            - "reader_f1": Average overlap between predicted answers and their corresponding correct answers
+            - "total_retrieve_time": Time retriever needed to retrieve documents for all questions
+            - "avg_retrieve_time": Average time needed to retrieve documents for one question
+            - "total_reader_time": Time reader needed to extract answer out of retrieved documents for all questions
+                                   where the correct document is among the retrieved ones
+            - "avg_reader_time": Average time needed to extract answer out of retrieved documents for one question
+            - "total_finder_time": Total time for whole pipeline
+
+        :param label_index: Elasticsearch index where labeled questions are stored
+        :type label_index: str
+        :param doc_index: Elasticsearch index where documents that are used for evaluation are stored
+        :type doc_index: str
+        :param top_k_retriever: How many documents per question to return and pass to reader
+        :type top_k_retriever: int
+        :param top_k_reader: How many answers to return per question
+        :type top_k_reader: int
+        """
+        finder_start_time = time.time()
+        # extract all questions for evaluation
+        filter = {"origin": label_origin}
+        questions = self.retriever.document_store.get_all_docs_in_index(index=label_index, filters=filter)
+
+        correct_retrievals = 0
+        correct_readings = 0
+        summed_avg_precision_retriever = 0
+        summed_avg_precision_reader = 0
+        exact_matches = 0
+        summed_f1 = 0
+        retrieve_times = []
+        read_times = []
+
+        # retrieve documents
+        questions_with_docs = []
+        retriever_start_time = time.time()
+        for q_idx, question in enumerate(questions):
+            question_string = question["_source"]["question"]
+            single_retrieve_start = time.time()
+            retrieved_docs = self.retriever.retrieve(question_string, top_k=top_k_retriever, index=doc_index)
+            retrieve_times.append(time.time() - single_retrieve_start)
+            for doc_idx, doc in enumerate(retrieved_docs):
+                # check if correct doc among retrieved docs
+                if doc.meta["doc_id"] == question["_source"]["doc_id"]:
+                    correct_retrievals += 1
+                    summed_avg_precision_retriever += 1 / (doc_idx + 1)
+                    questions_with_docs.append({
+                        "question": question,
+                        "docs": retrieved_docs,
+                        "correct_es_doc_id": doc.id})
+                    break
+        retriever_total_time = time.time() - retriever_start_time
+        number_of_questions = q_idx + 1
+
+        previous_top_k_per_candidate = self.reader.top_k_per_candidate
+        previous_return_no_answers = self.reader.return_no_answers
+        self.reader.top_k_per_candidate = 1
path: haystack/finder.py
text: @user1:
The top k per candidate is tricky with a farm reader and not correctly handled within haystack currently.

diff_hunk: @@ -41,12 +41,58 @@ def __init__(self, document_store: Type[BaseDocumentStore], custom_query: str =
         self.document_store = document_store
         self.custom_query = custom_query
 
-    def retrieve(self, query: str, filters: dict = None, top_k: int = 10) -> [Document]:
-        documents = self.document_store.query(query, filters, top_k, self.custom_query)
+    def retrieve(self, query: str, filters: dict = None, top_k: int = 10, index: str = None) -> [Document]:
+        if index is None:
+            index = self.document_store.index
+
+        documents = self.document_store.query(query, filters, top_k, self.custom_query, index)
         logger.info(f"Got {len(documents)} candidates from retriever")
 
         return documents
 
+    def eval(self, label_index: str = "feedback", doc_index: str = "eval_document", label_origin: str = "gold_label",
+             top_k: int = 10) -> dict:
+        """
+        Performs evaluation on the Retriever.
+        Retriever is evaluated based on whether it finds the correct document given the question string and at which
+        position in the ranking of documents the correct document is.
+
+        Returns a dict containing the following metrics:
+            - "recall": Proportion of questions for which correct document is among retrieved documents
+            - "mean avg precision": Mean of average precision for each question. Rewards retrievers that give relevant
+              documents a higher rank.
+
+        :param label_index: Elasticsearch index where labeled questions are stored
+        :param doc_index: Elasticsearch index where documents that are used for evaluation are stored
+        :param top_k: How many documents to return per question
+        """
+
+        # extract all questions for evaluation
+        filter = {"origin": label_origin}
+        questions = self.document_store.get_all_docs_in_index(index=label_index, filters=filter)
+
+        # calculate recall and mean-average-precision
+        correct_retrievals = 0
+        summed_avg_precision = 0
+        for q_idx, question in enumerate(questions):
+            question_string = question["_source"]["question"]
+            retrieved_docs = self.retrieve(question_string, top_k=top_k, index=doc_index)
+            # check if correct doc in retrieved docs
+            for doc_idx, doc in enumerate(retrieved_docs):
+                if doc.meta["doc_id"] == question["_source"]["doc_id"]:
+                    correct_retrievals += 1
+                    summed_avg_precision += 1 / (doc_idx + 1)
+                    break
+
+        number_of_questions = q_idx + 1
+        recall = correct_retrievals / number_of_questions
+        mean_avg_precision = summed_avg_precision / number_of_questions
+
+        logger.info((f"For {correct_retrievals} out of {number_of_questions} questions ({recall:.2%}), the answer was in"
+                     f" the top-{top_k} candidate passages selected by the retriever."))
+
+        return {"recall": recall, "mean avg precision": mean_avg_precision}
path: haystack/retriever/elasticsearch.py
text: @user1:
For consistency with the metrics from Finder.eval() we should name it "map"

diff_hunk: @@ -96,3 +98,164 @@ def get_answers_via_similar_questions(self, question: str, top_k_retriever: int
             results["answers"].append(cur_answer)
 
         return results
+
+    def eval(self, label_index: str = "feedback", doc_index: str = "eval_document", label_origin: str = "gold_label",
+             top_k_retriever: int = 10, top_k_reader: int = 10):
+        """
+        Evaluation of the whole pipeline by first evaluating the Retriever and then evaluating the Reader on the result
+        of the Retriever.
+
+        Returns a dict containing the following metrics:
+            - "retriever_recall": Proportion of questions for which correct document is among retrieved documents
+            - "retriever_map": Mean of average precision for each question. Rewards retrievers that give relevant
+              documents a higher rank.
+            - "reader_recall": Proportion of predicted answers that overlap with correct answer
+            - "reader_map": Reader re-ranks documents retrieved by Retriever. Mean average precision of the re-ranking.
path: haystack/finder.py
text: @user1:
Really nice, details docstrings! Just about the "reader_map" one: This might be confusing as the reader is not really re-ranking documents, right? It's more ranking the answers and they relate to documents. So maybe something like "Mean average precision of ranked answers by the Reader. Rewards readers that give correct answer(s) a high rank (mAP= 1 means all valid answers always on very top of list).

@user2:
Bogdan and me discussed this - a simple intuition for the user is to say the reader "re-ranks" retriever results, which is of course not the whole story for extractive QA. Also see line 186f where we actual count re ranking of pure docs in the reader_map. So lets either change this reader_map or increase complexity in the documentation like:

Reader re-ranks documents returned by Retriever. Mean average precision of the re-ranking. In case of extractive Readers the actual answer is compared to the extracted prediction. So in this case the Reader is not just re-ranking documents but comparing label and prediction on a different scale than the retriever.