Repository: deepset-ai/haystack

Issue:
Proper Probabilities for Answers
At the moment, we have pseudo probabilities in Haystack. Proper probabilities would be helpful to decide if it makes sense to show an answer to the user. If an answer has nothing to do with the question, the users are confused or even doubt that the technology is working. 


Discussion:
Totally, we need better and more reliable scores or even probabilities.

@julian-risch and me will be working on model confidence, that then results in better scores, the coming weeks. I think we will  tackle this in FARM, since it is a QA Modelling related task.
> will be working on model confidence, that then results in better scores,

Hey @Timoeller I would like to learn this so I am happy to contribute if given direction.
Hey Lalit, thanks for the kind offer. This is actually a bigger topic that we started to plan and will tackle in the coming weeks.

We could definitely use your support and feedback on related the code we will write. 
Once we have laid out our detailed plan in the linked FARM issue we can also discuss if there are smaller tasks for you to work on. 
We can also have a call where we discuss the solution in general, I know you are very interested in the NLP aspects of haystack and the model confidence would be a good way to show you QA modelling related details.
What do you say?  
Sure make sense. Ping me when you start working on this topic.

Excited to work with @julian-risch, number of publications in his PhD are beyond my imagination. Specially topic related to abusive language detection.

Resolving patch:
diff --git a/haystack/reader/farm.py b/haystack/reader/farm.py
--- a/haystack/reader/farm.py
+++ b/haystack/reader/farm.py
@@ -559,7 +559,7 @@ def _extract_answers_of_predictions(self, predictions: List[QAPred], top_k: Opti
                         "answer": ans.answer,
                         "score": ans.score,
                         # just a pseudo prob for now
-                        "probability": self._get_pseudo_prob(ans.score),
+                        "probability": ans.confidence,
                         "context": ans.context_window,
                         "offset_start": ans.offset_answer_start - ans.offset_context_window_start,
                         "offset_end": ans.offset_answer_end - ans.offset_context_window_start,
diff --git a/haystack/retriever/dense.py b/haystack/retriever/dense.py
--- a/haystack/retriever/dense.py
+++ b/haystack/retriever/dense.py
@@ -37,6 +37,7 @@ def __init__(self,
                  document_store: BaseDocumentStore,
                  query_embedding_model: Union[Path, str] = "facebook/dpr-question_encoder-single-nq-base",
                  passage_embedding_model: Union[Path, str] = "facebook/dpr-ctx_encoder-single-nq-base",
+                 single_model_path: Optional[Union[Path, str]] = None,
                  model_version: Optional[str] = None,
                  max_seq_len_query: int = 64,
                  max_seq_len_passage: int = 256,
@@ -73,6 +74,9 @@ def __init__(self,
         :param passage_embedding_model: Local path or remote name of passage encoder checkpoint. The format equals the
                                         one used by hugging-face transformers' modelhub models
                                         Currently available remote names: ``"facebook/dpr-ctx_encoder-single-nq-base"``
+        :param single_model_path: Local path or remote name of a query and passage embedder in one single model. Those
+                                  models are typically trained within FARM.
+                                  Currently available remote names: TODO add FARM DPR model to HF modelhub
         :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.
         :param max_seq_len_query: Longest length of each query sequence. Maximum number of tokens for the query text. Longer ones will be cut down."
         :param max_seq_len_passage: Longest length of each passage/context sequence. Maximum number of tokens for the passage text. Longer ones will be cut down."
@@ -96,8 +100,6 @@ def __init__(self,
 
         self.document_store = document_store
         self.batch_size = batch_size
-        self.max_seq_len_passage = max_seq_len_passage
-        self.max_seq_len_query = max_seq_len_query
         self.progress_bar = progress_bar
         self.top_k = top_k
 
@@ -115,7 +117,6 @@ def __init__(self,
         else:
             self.device = torch.device("cpu")
 
-        self.embed_title = embed_title
         self.infer_tokenizer_classes = infer_tokenizer_classes
         tokenizers_default_classes = {
             "query": "DPRQuestionEncoderTokenizer",
@@ -126,43 +127,52 @@ def __init__(self,
             tokenizers_default_classes["passage"] = None # type: ignore
 
         # Init & Load Encoders
-        self.query_tokenizer = Tokenizer.load(pretrained_model_name_or_path=query_embedding_model,
-                                              revision=model_version,
-                                              do_lower_case=True,
-                                              use_fast=use_fast_tokenizers,
-                                              tokenizer_class=tokenizers_default_classes["query"])
-        self.query_encoder = LanguageModel.load(pretrained_model_name_or_path=query_embedding_model,
-                                                revision=model_version,
-                                                language_model_class="DPRQuestionEncoder")
-        self.passage_tokenizer = Tokenizer.load(pretrained_model_name_or_path=passage_embedding_model,
-                                                revision=model_version,
-                                                do_lower_case=True,
-                                                use_fast=use_fast_tokenizers,
-                                                tokenizer_class=tokenizers_default_classes["passage"])
-        self.passage_encoder = LanguageModel.load(pretrained_model_name_or_path=passage_embedding_model,
+        if single_model_path is None:
+            self.query_tokenizer = Tokenizer.load(pretrained_model_name_or_path=query_embedding_model,
                                                   revision=model_version,
-                                                  language_model_class="DPRContextEncoder")
-
-        self.processor = TextSimilarityProcessor(tokenizer=self.query_tokenizer,
-                                                 passage_tokenizer=self.passage_tokenizer,
-                                                 max_seq_len_passage=self.max_seq_len_passage,
-                                                 max_seq_len_query=self.max_seq_len_query,
-                                                 label_list=["hard_negative", "positive"],
-                                                 metric="text_similarity_metric",
-                                                 embed_title=self.embed_title,
-                                                 num_hard_negatives=0,
-                                                 num_positives=1)
-
-        prediction_head = TextSimilarityHead(similarity_function=similarity_function)
-        self.model = BiAdaptiveModel(
-            language_model1=self.query_encoder,
-            language_model2=self.passage_encoder,
-            prediction_heads=[prediction_head],
-            embeds_dropout_prob=0.1,
-            lm1_output_types=["per_sequence"],
-            lm2_output_types=["per_sequence"],
-            device=self.device,
-        )
+                                                  do_lower_case=True,
+                                                  use_fast=use_fast_tokenizers,
+                                                  tokenizer_class=tokenizers_default_classes["query"])
+            self.query_encoder = LanguageModel.load(pretrained_model_name_or_path=query_embedding_model,
+                                                    revision=model_version,
+                                                    language_model_class="DPRQuestionEncoder")
+            self.passage_tokenizer = Tokenizer.load(pretrained_model_name_or_path=passage_embedding_model,
+                                                    revision=model_version,
+                                                    do_lower_case=True,
+                                                    use_fast=use_fast_tokenizers,
+                                                    tokenizer_class=tokenizers_default_classes["passage"])
+            self.passage_encoder = LanguageModel.load(pretrained_model_name_or_path=passage_embedding_model,
+                                                      revision=model_version,
+                                                      language_model_class="DPRContextEncoder")
+
+            self.processor = TextSimilarityProcessor(query_tokenizer=self.query_tokenizer,
+                                                     passage_tokenizer=self.passage_tokenizer,
+                                                     max_seq_len_passage=max_seq_len_passage,
+                                                     max_seq_len_query=max_seq_len_query,
+                                                     label_list=["hard_negative", "positive"],
+                                                     metric="text_similarity_metric",
+                                                     embed_title=embed_title,
+                                                     num_hard_negatives=0,
+                                                     num_positives=1)
+            prediction_head = TextSimilarityHead(similarity_function=similarity_function)
+            self.model = BiAdaptiveModel(
+                language_model1=self.query_encoder,
+                language_model2=self.passage_encoder,
+                prediction_heads=[prediction_head],
+                embeds_dropout_prob=0.1,
+                lm1_output_types=["per_sequence"],
+                lm2_output_types=["per_sequence"],
+                device=self.device,
+            )
+        else:
+            self.processor = TextSimilarityProcessor.load_from_dir(single_model_path)
+            self.processor.max_seq_len_passage = max_seq_len_passage
+            self.processor.max_seq_len_query = max_seq_len_query
+            self.processor.embed_title = embed_title
+            self.processor.num_hard_negatives = 0
+            self.processor.num_positives = 1  # during indexing of documents only one embedding is created
+            self.model = BiAdaptiveModel.load(single_model_path, device=self.device)
+
         self.model.connect_heads_with_processor(self.processor.tasks, require_labels=False)
 
     def retrieve(self, query: str, filters: dict = None, top_k: Optional[int] = None, index: str = None) -> List[Document]:
@@ -318,21 +328,14 @@ def train(self,
         :param passage_encoder_save_dir: directory inside save_dir where passage_encoder model files are saved
         """
 
-        self.embed_title = embed_title
-        self.processor = TextSimilarityProcessor(tokenizer=self.query_tokenizer,
-                                                 passage_tokenizer=self.passage_tokenizer,
-                                                 max_seq_len_passage=self.max_seq_len_passage,
-                                                 max_seq_len_query=self.max_seq_len_query,
-                                                 label_list=["hard_negative", "positive"],
-                                                 metric="text_similarity_metric",
-                                                 data_dir=data_dir,
-                                                 train_filename=train_filename,
-                                                 dev_filename=dev_filename,
-                                                 test_filename=test_filename,
-                                                 dev_split=dev_split,
-                                                 embed_title=self.embed_title,
-                                                 num_hard_negatives=num_hard_negatives,
-                                                 num_positives=num_positives)
+        self.processor.embed_title = embed_title
+        self.processor.data_dir = data_dir
+        self.processor.train_filename = train_filename
+        self.processor.dev_filename = dev_filename
+        self.processor.test_filename = test_filename
+        self.processor.dev_split = dev_split
+        self.processor.num_hard_negatives = num_hard_negatives
+        self.processor.num_positives = num_positives
 
         self.model.connect_heads_with_processor(self.processor.tasks, require_labels=True)
 


Test patch:
diff --git a/test/test_retriever.py b/test/test_retriever.py
--- a/test/test_retriever.py
+++ b/test/test_retriever.py
@@ -194,10 +194,10 @@ def sum_params(model):
     #     assert (p1.data.ne(p2.data).sum() == 0)
 
     # attributes
-    assert loaded_retriever.embed_title == True
+    assert loaded_retriever.processor.embed_title == True
     assert loaded_retriever.batch_size == 16
-    assert loaded_retriever.max_seq_len_passage == 256
-    assert loaded_retriever.max_seq_len_query == 64
+    assert loaded_retriever.processor.max_seq_len_passage == 256
+    assert loaded_retriever.processor.max_seq_len_query == 64
 
     # Tokenizer
     assert isinstance(loaded_retriever.passage_tokenizer, DPRContextEncoderTokenizerFast)