Repository: deepset-ai/haystack

Issue:
Make FAISSDocumentStore work with yaml format (used by REST API folder)
**Is your feature request related to a problem? Please describe.**
I'm trying to get the DensePassageRetriever (DPR) working with a FAISSDocumentStore and into a REST API using the rest_api/ folder in the haystack repo. This rest_api/ folder requires me to use the yaml formatting. I use the code in the [DPR tutorial](https://haystack.deepset.ai/tutorials/dense-passage-retrieval) to load the FAISSDocumentStore and check that the ExtractiveQAPipeline is working when I run it in the same python file as the FAISSDocumentStore loading. However, when I then use the following in my rest_api/pipeline/pipelines.yaml file, I get empty results from the corresponding extractive QA pipeline I put together in that file.
```
- name: FAISSDocumentStore
  type: FAISSDocumentStore
  params:
    sql_url: "sqlite:///faiss_document_store.db"
```
It is because it is not detecting the FAISSDocumentStore index. I know it is because it is not detecting the FAISSDocumentStore index because I'm able to get it working in Python using the `save` and `load` methods described [here](https://haystack.deepset.ai/components/document-store):

![Screen Shot 2021-11-04 at 11 19 36 PM](https://user-images.githubusercontent.com/8754873/140529755-ddb4fbeb-f791-40ba-ab77-fab8023d09cc.png)

**Describe the solution you'd like**
I request a new argument and corresponding loading functionality in FAISSDocumentStore's constructor, so that I can get the equivalent of the `load` method's functionality using an argument to FAISSDocumentStore's constructor. Then, I'd just pass the index files name in for that argument in the yaml file.

**Describe alternatives you've considered**
I tried editing the faiss.py constructor myself to just add the following:
```
if load_index is not None:
    self = FAISSDocumentStore.load(load_index)
else:
    <previous constructor code>
```
but this doesn't work, as you might be able to tell from just looking at it haha.



Discussion:
Hello @bradyneal, thank you for this bug report! f I understand your description correctly:
- You have a populated FAISS document store
- You previously called `document_store.save("my_faiss_index.faiss")` from Python and it worked
- You verified that you can load it back in Python with `FAISSDocumentStore.load("my_faiss_index.faiss")`
- But when using a `pipeline.yaml` that contains the snippet you posted above, the doc store is empty.

I'm sure you verified already, but while I try to replicate your issue. please double-check that:

1) the index actually exists
2) you have no typos in your `pipeline.yaml`
3) the REST API points to the correct `pipeline.yaml` file
4) the startup logs contain no unusual warnings. If you spot any, please share a dump of all the logs produced, from startup.

I'm going to be back here soon to let you know if I can replicate the issue and what to do next :slightly_smiling_face: 
Yes, that all sounds right 👍 
So, I didn't manage to replicate your issue exactly, but I spotted a potential issue.

In your snippet, the node name is `FAISSDocumentStore`. If the rest of the `pipeline.yaml` file is untouched, then it won't work: the rest of the nodes refer to a node called only `DocumentStore`, so you have to change the definition of your FAISS doc store to:
```yaml
- name: DocumentStore    # Note that I changed this line
  type: FAISSDocumentStore
  params:
    sql_url: "sqlite:///faiss_document_store.db"
```
However, with your original snippet my application crashes rather than loading an empty document store, so something else might be up. If the snippet above does not help you, please share:

- The entire `pipeline.yaml` file
- The logs of your application during startup

Thanks for volunteering the in-depth help @ZanSara; very awesome. Let me give a bit more context before I go recreate these issues on our server (we switched away from using haystack's rest_api folder and changed to our own API code because we were getting slowed down by too many issues we were having with the rest_api folder).

More context: I created this issue after @brandenchan told me the following [on Slack](https://haystack-community.slack.com/archives/C01J3TZM9HT/p1636082711125400):
> That's actually a good point. I don't believe that its possible right now to load a saved FAISS Document Store. You would have to run a predefined indexing pipeline via REST API in order to populate the FAISS Document Store.
[...] In any case, I think it would make sense to add support for loading a saved FAISS Document Store from YAML. Would you like to open an issue for this?

Basically, the functionality I was looking for, at the time, was this:
**access to the FAISSDocumentStore.load() functionality in the yaml format**.

Branden told me that haystack does not yet have this functionality, so I posted this issue. However, maybe haystack already does have this functionality. If so, my GItHub issue is as simple as the following question:
**How can I get the FAISSDocumentStore.load() functionality in the yaml format?**
Hey @bradyneal, we will pick this up in the next sprint. From a quick look, it might already be enough to add the params `faiss_index_path` and `faiss_config_path` to the init of `FAISSDocumentstore` to allow loading an existing index from disk. These params could then also be specified in the yaml. 

@tstadel We should make sure to add a few checks to make sure that no invalid parameter combinations are supplied at init (e.g. faiss_index and faiss_index_path are exclusive) 

Resolving patch:
diff --git a/haystack/document_stores/faiss.py b/haystack/document_stores/faiss.py
--- a/haystack/document_stores/faiss.py
+++ b/haystack/document_stores/faiss.py
@@ -18,6 +18,7 @@
 from haystack.schema import Document
 from haystack.document_stores.sql import SQLDocumentStore
 from haystack.document_stores.base import get_batches_from_generator
+from inspect import Signature, signature
 
 
 logger = logging.getLogger(__name__)
@@ -45,6 +46,8 @@ def __init__(
         embedding_field: str = "embedding",
         progress_bar: bool = True,
         duplicate_documents: str = 'overwrite',
+        faiss_index_path: Union[str, Path] = None,
+        faiss_config_path: Union[str, Path] = None,
         **kwargs,
     ):
         """
@@ -84,7 +87,20 @@ def __init__(
                                     overwrite: Update any existing documents with the same ID when adding documents.
                                     fail: an error is raised if the document ID of the document being added already
                                     exists.
+        :param faiss_index_path: Stored FAISS index file. Can be created via calling `save()`.
+            If specified no other params besides faiss_config_path must be specified.
+        :param faiss_config_path: Stored FAISS initial configuration parameters. 
+            Can be created via calling `save()`
         """
+        # special case if we want to load an existing index from disk
+        # load init params from disk and run init again
+        if faiss_index_path is not None:
+            sig = signature(self.__class__.__init__)
+            self._validate_params_load_from_disk(sig, locals(), kwargs)
+            init_params = self._load_init_params_from_config(faiss_index_path, faiss_config_path)
+            self.__class__.__init__(self, **init_params)
+            return
+
         # save init parameters to enable export of component config as YAML
         self.set_config(
             sql_url=sql_url, 
@@ -133,6 +149,30 @@ def __init__(
             index=index
         )
 
+        self._validate_index_sync()
+
+    def _validate_params_load_from_disk(self, sig: Signature, locals: dict, kwargs: dict):
+        allowed_params = ["faiss_index_path", "faiss_config_path", "self", "kwargs"]
+        invalid_param_set = False
+
+        for param in sig.parameters.values():
+            if param.name not in allowed_params and param.default != locals[param.name]:
+                    invalid_param_set = True
+                    break
+        
+        if invalid_param_set or len(kwargs) > 0:
+            raise ValueError("if faiss_index_path is passed no other params besides faiss_config_path are allowed.")
+
+    def _validate_index_sync(self):        
+        # This check ensures the correct document database was loaded.
+        # If it fails, make sure you provided the path to the database
+        # used when creating the original FAISS index
+        if not self.get_document_count() == self.get_embedding_count():
+            raise ValueError("The number of documents present in the SQL database does not "
+                             "match the number of embeddings in FAISS. Make sure your FAISS "
+                             "configuration file correctly points to the same database that "
+                             "was used when creating the original index.")
+
     def _create_new_index(self, vector_dim: int, metric_type, index_factory: str = "Flat", **kwargs):
         if index_factory == "HNSW":
             # faiss index factory doesn't give the same results for HNSW IP, therefore direct init.
@@ -489,23 +529,7 @@ def save(self, index_path: Union[str, Path], config_path: Optional[Union[str, Pa
         with open(config_path, 'w') as ipp:
             json.dump(self.pipeline_config["params"], ipp)
 
-    @classmethod
-    def load(cls, index_path: Union[str, Path], config_path: Optional[Union[str, Path]] = None):
-        """
-        Load a saved FAISS index from a file and connect to the SQL database.
-        Note: In order to have a correct mapping from FAISS to SQL,
-              make sure to use the same SQL DB that you used when calling `save()`.
-
-        :param index_path: Stored FAISS index file. Can be created via calling `save()`
-        :param config_path: Stored FAISS initial configuration parameters. 
-            Can be created via calling `save()`
-        :param sql_url: Connection string to the SQL database that contains your docs and metadata.
-            Overrides the value defined in the `faiss_init_params_path` file, if present
-        :param index: Index name to load the FAISS index as. It must match the index name used for
-                      when creating the FAISS index. Overrides the value defined in the 
-                      `faiss_init_params_path` file, if present
-        :return: the DocumentStore
-        """
+    def _load_init_params_from_config(self, index_path: Union[str, Path], config_path: Optional[Union[str, Path]] = None):
         if not config_path:
             index_path = Path(index_path)
             config_path = index_path.with_suffix(".json")
@@ -523,17 +547,19 @@ def load(cls, index_path: Union[str, Path], config_path: Optional[Union[str, Pat
 
         # Add other init params to override the ones defined in the init params file
         init_params["faiss_index"] = faiss_index
-        init_params["vector_dim"]=faiss_index.d
+        init_params["vector_dim"] = faiss_index.d
 
-        document_store = cls(**init_params)
+        return init_params
 
-        # This check ensures the correct document database was loaded.
-        # If it fails, make sure you provided the path to the database
-        # used when creating the original FAISS index
-        if not document_store.get_document_count() == document_store.get_embedding_count():
-            raise ValueError("The number of documents present in the SQL database does not "
-                             "match the number of embeddings in FAISS. Make sure your FAISS "
-                             "configuration file correctly points to the same database that "
-                             "was used when creating the original index.")
+    @classmethod
+    def load(cls, index_path: Union[str, Path], config_path: Optional[Union[str, Path]] = None):
+        """
+        Load a saved FAISS index from a file and connect to the SQL database.
+        Note: In order to have a correct mapping from FAISS to SQL,
+              make sure to use the same SQL DB that you used when calling `save()`.
 
-        return document_store
+        :param index_path: Stored FAISS index file. Can be created via calling `save()`
+        :param config_path: Stored FAISS initial configuration parameters. 
+            Can be created via calling `save()`
+        """
+        return cls(faiss_index_path=index_path, faiss_config_path=config_path)


Test patch:
diff --git a/test/samples/pipeline/test_pipeline_faiss_indexing.yaml b/test/samples/pipeline/test_pipeline_faiss_indexing.yaml
new file mode 100644
--- /dev/null
+++ b/test/samples/pipeline/test_pipeline_faiss_indexing.yaml
@@ -0,0 +1,31 @@
+version: '0.7'
+
+components:
+  - name: DPRRetriever
+    type: DensePassageRetriever
+    params:
+      document_store: NewFAISSDocumentStore
+  - name: PDFConverter
+    type: PDFToTextConverter
+    params:
+      remove_numeric_tables: false
+  - name: Preprocessor
+    type: PreProcessor
+    params:
+      clean_whitespace: true
+  - name: NewFAISSDocumentStore
+    type: FAISSDocumentStore
+
+
+pipelines:
+  - name: indexing_pipeline
+    type: Pipeline
+    nodes:
+      - name: PDFConverter
+        inputs: [File]
+      - name: Preprocessor
+        inputs: [PDFConverter]
+      - name: DPRRetriever
+        inputs: [Preprocessor]
+      - name: NewFAISSDocumentStore
+        inputs: [DPRRetriever]
diff --git a/test/samples/pipeline/test_pipeline_faiss_retrieval.yaml b/test/samples/pipeline/test_pipeline_faiss_retrieval.yaml
new file mode 100644
--- /dev/null
+++ b/test/samples/pipeline/test_pipeline_faiss_retrieval.yaml
@@ -0,0 +1,19 @@
+version: '0.7'
+
+components:
+  - name: DPRRetriever
+    type: DensePassageRetriever
+    params:
+      document_store: ExistingFAISSDocumentStore
+  - name: ExistingFAISSDocumentStore
+    type: FAISSDocumentStore
+    params:
+      faiss_index_path: 'existing_faiss_document_store'
+
+
+pipelines:
+  - name: query_pipeline
+    type: Pipeline
+    nodes:
+      - name: DPRRetriever
+        inputs: [Query]
diff --git a/test/test_faiss_and_milvus.py b/test/test_faiss_and_milvus.py
--- a/test/test_faiss_and_milvus.py
+++ b/test/test_faiss_and_milvus.py
@@ -51,6 +51,16 @@ def test_faiss_index_save_and_load(tmp_path):
     # Check if the init parameters are kept
     assert not new_document_store.progress_bar
 
+    # test loading the index via init
+    new_document_store = FAISSDocumentStore(faiss_index_path=tmp_path / "haystack_test_faiss")
+
+    # check faiss index is restored
+    assert new_document_store.faiss_indexes[document_store.index].ntotal == len(DOCUMENTS)
+    # check if documents are restored
+    assert len(new_document_store.get_all_documents()) == len(DOCUMENTS)
+    # Check if the init parameters are kept
+    assert not new_document_store.progress_bar
+
 
 @pytest.mark.skipif(sys.platform in ['win32', 'cygwin'], reason="Test with tmp_path not working on windows runner")
 def test_faiss_index_save_and_load_custom_path(tmp_path):
@@ -80,6 +90,31 @@ def test_faiss_index_save_and_load_custom_path(tmp_path):
     # Check if the init parameters are kept
     assert not new_document_store.progress_bar
 
+    # test loading the index via init
+    new_document_store = FAISSDocumentStore(faiss_index_path=tmp_path / "haystack_test_faiss", faiss_config_path=tmp_path / "custom_path.json")
+
+    # check faiss index is restored
+    assert new_document_store.faiss_indexes[document_store.index].ntotal == len(DOCUMENTS)
+    # check if documents are restored
+    assert len(new_document_store.get_all_documents()) == len(DOCUMENTS)
+    # Check if the init parameters are kept
+    assert not new_document_store.progress_bar
+
+
+@pytest.mark.skipif(sys.platform in ['win32', 'cygwin'], reason="Test with tmp_path not working on windows runner")
+def test_faiss_index_mutual_exclusive_args(tmp_path):
+    with pytest.raises(ValueError):
+        FAISSDocumentStore(
+            sql_url=f"sqlite:////{tmp_path/'haystack_test.db'}",
+            faiss_index_path=f"{tmp_path/'haystack_test'}"
+        )
+
+    with pytest.raises(ValueError):
+        FAISSDocumentStore(
+            f"sqlite:////{tmp_path/'haystack_test.db'}",
+            faiss_index_path=f"{tmp_path/'haystack_test'}"
+        )
+
 
 @pytest.mark.parametrize("document_store", ["faiss"], indirect=True)
 @pytest.mark.parametrize("index_buffer_size", [10_000, 2])
diff --git a/test/test_pipeline.py b/test/test_pipeline.py
--- a/test/test_pipeline.py
+++ b/test/test_pipeline.py
@@ -1,5 +1,6 @@
 from pathlib import Path
 
+import os
 import json
 import math
 import pytest
@@ -747,3 +748,43 @@ def test_query_pipeline_with_document_classifier(document_store):
     assert prediction["answers"][0].meta["classification"]["label"] == "joy"
     assert "_debug" not in prediction.keys()
 
+
+def test_existing_faiss_document_store():
+    clean_faiss_document_store()
+
+    pipeline = Pipeline.load_from_yaml(
+        Path(__file__).parent/"samples"/"pipeline"/"test_pipeline_faiss_indexing.yaml", pipeline_name="indexing_pipeline"
+    )
+    pipeline.run(
+        file_paths=Path(__file__).parent/"samples"/"pdf"/"sample_pdf_1.pdf"
+    )
+
+    new_document_store = pipeline.get_document_store()
+    new_document_store.save('existing_faiss_document_store')
+
+    # test correct load of query pipeline from yaml
+    pipeline = Pipeline.load_from_yaml(
+        Path(__file__).parent/"samples"/"pipeline"/"test_pipeline_faiss_retrieval.yaml", pipeline_name="query_pipeline"
+    )
+
+    retriever = pipeline.get_node("DPRRetriever")
+    existing_document_store = retriever.document_store
+    faiss_index = existing_document_store.faiss_indexes['document']
+    assert faiss_index.ntotal == 2
+
+    prediction = pipeline.run(
+        query="Who made the PDF specification?", params={"DPRRetriever": {"top_k": 10}}
+    )
+
+    assert prediction["query"] == "Who made the PDF specification?"
+    assert len(prediction["documents"]) == 2
+    clean_faiss_document_store()
+
+
+def clean_faiss_document_store():
+    if Path('existing_faiss_document_store').exists():
+        os.remove('existing_faiss_document_store')
+    if Path('existing_faiss_document_store.json').exists():
+        os.remove('existing_faiss_document_store.json')
+    if Path('faiss_document_store.db').exists():
+        os.remove('faiss_document_store.db')