Repository: deepset-ai/haystack

Issue:
How to save embedding_model and document_store locally ?
I wonder if it is possible to do something similar to `reader.save(directory="my_path")` but for a `EmbeddingRetriever`


Discussion:
Hey @arthurbarros 
the retrievers are not trained (for now) so there is not much need to store them. 
The underlying model like "deepset/sentence_bert" is downloaded and cached when first calling the EmbeddingRetriever, so you only download it once.

Though saving the Retrievers might be an interesting feature if you are in an offline setting or copy the model between machines - working with the cache files directly is rather cumbersome. Could you elaborate why you need this functionality, so we can better prioritize this feature?


Closing this now. Feel free to reopen
Hi @Timoeller,
I have used dpr retriever on colab using gpu (following tute 6) for qa on my documents, I want to use this model on my local (or instance) without a gpu. `document_store.update_embedding(retriever) #document_store: FAISSDocumentStore, retriever: DensePassageRetriever` on instance is taking much longer time (it is low config machine). This step needs to be run only once. If I could save retriever, or document_store (with its updated embeddings) and copy to my instance, I have to only load them without running bottleneck step and every time, demo would be fast.
am I doing something wrong?

also couldn't use `document_store.save("filepath")` because `FAISSDocumentStore.load()` needs sql_url, why don't we need sql_url during initialization but it is essential during loading?
I'm also wondering how to save the document store once we've taken the time to generate the embeddings and loading them. 
Hey @Weilin37, 

What type of document store are you using? 

If you use ElasticsearchDocumentStore the data will be automatically persisted by elasticsearch. 

If you use FAISS, the embeddings are stored in a FAISS Index. You can save that index via `FAISSDocumentStore.save("file_path")`. All remaining data (e.g the document text and metadata) is automatically kept persistent in SQL. You can load it back via `FAISSDocumentStore.load(faiss_file_path="file_path", sql_url= "sqlite:///")`

Hope this helps!
Thanks! This is definitely the answer I was looking for! (I am using FAISS)
Hi @tholor,

Just curious if that also applies for document_store that already had update_embeddings applied? Using Faiss by the way.
I did try your example but when I loaded it, the embeddings where empty. I haven't tried all examples in the documentation yet so maybe im missing something
Yes, absolutely. 
So I was trying this out in colab 
document_store = FAISSDocumentStore()
document_store.write_documents(datadict)
retriever = EmbeddingRetriever(document_store=document_store,
                               embedding_model="deepset/sentence_bert",
                               use_gpu=False)
							   
document_store.update_embeddings(retriever)
document_store.save("testfile_path")
document_store2 = FAISSDocumentStore()
document_store2.load(faiss_file_path="testfile_path", sql_url= "sqlite:///")

but document_store2 always comes up empty. Please guide me where I went wrong. I think its in the sql part but I am not also familiar with it
Can you please try:

``` 
document_store = FAISSDocumentStore(sql_url= "sqlite:///haystack_test_faiss.db")
document_store.write_documents(datadict)
retriever = EmbeddingRetriever(document_store=document_store,
embedding_model="deepset/sentence_bert", use_gpu=False)
document_store.update_embeddings(retriever)
document_store.save("testfile_path")
document_store2 = FAISSDocumentStore.load(faiss_file_path="testfile_path", sql_url= "sqlite:///haystack_test_faiss.db")
``` 
It worked! Thank you.

Also I noticed that after applying 'update_embedding', when I try to check the embedding value of items via 'get_all_documents' within the document_store, embedding values remain empty. Is it the expected behavior or am I misunderstanding how update_embedding works?
You are totally right, the embeddings should not remain empty. @tanaysoni can you please check this and verify that saving/loading is working as expected?
Hi @AzureAlph, currently, the `FAISSDocumentStore` does not return embeddings with the documents by default.

With #615, an option to get the embeddings using the `return_embedding` parameter would get added to the `FAISSDocumentStore.get_all_documents()`.

In Google Colab I have used the below code to update the embedding and save the document_store and the retriever.
```
document_store = FAISSDocumentStore(faiss_index_factory_str="Flat", sql_url= "sqlite:///haystack_test_faiss.db")
from haystack.retriever.dense import DensePassageRetriever
retriever = DensePassageRetriever(document_store=document_store,
                                  query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
                                  passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
                                  max_seq_len_query=64,
                                  max_seq_len_passage=256,
                                  batch_size=16,
                                  use_gpu=True,
                                  embed_title=True,
                                  use_fast_tokenizers=True)
document_store.update_embeddings(retriever)
retriever.save("retriever.pt")
document_store.save("faiss.index")
```
Later in my local machine, I have the following snippet which loads the document store and the saved retriever
```
document_store = FAISSDocumentStore.load("faiss.index",sql_url="sqlite:///haystack_test_faiss.db", index = None)
retriever = DensePassageRetriever.load("retriever.pt", document_store=document_store)
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True)
finder = Finder(reader, retriever)
prediction = finder.get_answers(question="Who is the sister of Sansa?", top_k_retriever=10, top_k_reader=5)
```
But This returns 0 candidates by the retriever. Even though the same was answered in Google Colab

Am I loading the files wrong?
@tholor I am experiencing the same problem as above. Is there a solution? I'm wondering if setting index to None has anything to do with it. 
Hey @benscottie thanks for bringing this up here. Could you open a separate issue with a more detailed explanation of your setup? I could imaging that different faiss versions for saving and loading might complicate it.
Or do you save and load in one colab environment?
hey @Timoeller thanks for your response. I am saving and loading in one environment and having the issue. Ultimately I would like to be able to save and load in separate environments. I will open a separate issue
> document_store2 = FAISSDocumentStore.load(faiss_file_path="testfile_path", sql_url= "sqlite:///haystack_test_faiss.db")

Hi, does your example mean that, if sql_url was not specified from the start, documents are not persisted anywhere and it's not possible to load them gain? If I'm wrong, where can I find them? According to the docs, default path should be "sqlite:///faiss_document_store.db", but I can't find such file, faiss_document_store.db, anywhere on my PC.
Hey @fingoldo, 
We recently changed the save/load behaviour for FAISS (see https://github.com/deepset-ai/haystack/pull/1459). With the latest master version, you should indeed have the default `sqlite:///faiss_document_store.db` and find a file accordingly on your system. This was not the case in older haystack versions where we used the inmemory based sqllite as default.

You still need to call `FAISSDocumentStore.save()` to save also the FAISS index and the configuration. Just having the SQL DB won't be enough to restore your doc store. 

Hope this helps. 


Thank you very much for the clarification... I have installed farm-haystack recently, a few days ago. After reading ~2M small documents, updating embeddings and calling .save(), I can't find the .db file anywhere, but the RAM footprint is 18Gbs. So probably I had an earlier version somehow which is in-memory SQL. I guess the only option I have is to reinstall the haystack & reindex from scratch...
Yes, the above PR was just merged 4 days ago (so after our latest 0.10. pypy release). I'd suggest you re-install directly from master via: 
```
git clone https://github.com/deepset-ai/haystack.git
cd haystack
pip install --editable .
```
I apologize,  is changing of vector_dim of a FAISS store supported, without reimporting all the documents? For example, when I want to test many models to see which one gives the most sensible results. 'cause I tried applying "sentence-transformers/all-MiniLM-L6-v2" and it turned out to be a mismatch between default vector_dim  of the store (768) and the actual dimension size of the retriever's model (384. is there some standard way to query this number from the retriever, by the way?). When I changed FAISS_store.vector_dim = 384 and tried update_embeddings() again, it was the same error, so I guess changing is not supported at all or must be done at store init step.
Also, it seems that you have a small inaccuracy in the code.

file:
haystack\haystack\document_store\faiss.py

method:
def load(cls, index_path: Union[str, Path], config_path: Optional[Union[str, Path]] = None):

code:

```python
        if not config_path:
            index_path = Path(index_path)
            faiss_init_params_path = index_path.with_suffix(".json")

        init_params: dict = {}
        try:
            with open(faiss_init_params_path, 'r') as ipp:
                init_params = json.load(ipp)
        except OSError as e:
```
if I provide config_path, load fails as faiss_init_params_path is not initialized.
> is changing of vector_dim of a FAISS store supported, without reimporting all the documents? 

No, that's currently not supported as FAISS doesn't allow any such permutations on an existing index. Most transformers use 768 as the dim - however, I see your point when switching to smaller retriever models. 
I think we can try to find a way to delete the FAISS index under the hood and repopulate it with embeddings from the new retriever using the documents that are still stored in SQL. Created a new issue for this: https://github.com/deepset-ai/haystack/issues/1505
Let me know if you are interested in contributing here.


> if I provide config_path, load fails as faiss_init_params_path is not initialized.

@ZanSara Can you please check?
> t transformers use 768 as the dim - however, I see your point when switching to smaller retriever models.
> I think we can try to find a way to delete the FAISS index under the hood and repopulate it with embeddi

Will things change if I switch to Milvus? I mean, that "index" parameter in many of the doc stores. Can it be used somehow to achieve that goal of testing multiple retrievers (and, therefore, embedding models) on the same document set?
> > t transformers use 768 as the dim - however, I see your point when switching to smaller retriever models.
> > I think we can try to find a way to delete the FAISS index under the hood and repopulate it with embeddi
> 
> Will things change if I switch to Milvus? I mean, that "index" parameter in many of the doc stores. Can it be used somehow to achieve that goal of testing multiple retrievers (and, therefore, embedding models) on the same document set?

I'd be willing to contribute something useful, but I'm not experienced at all in contributing (
> I'd be willing to contribute something useful, but I'm not experienced at all in contributing (

How about you give it a shot and if you don't succeed, we can still take over in a couple of weeks?
I added some first thoughts on the design [here](https://github.com/deepset-ai/haystack/issues/1505).

You can find some tips for contributing to Haystack [here](https://github.com/deepset-ai/haystack/blob/master/CONTRIBUTING.md). In short: If you create an early "work in progress" PR, we can also try to help and give feedback. We recommend using our CI for running the tests until #1353 is finished (which will simplify running local tests).
@tholor @fingoldo 
I have bit hacky solution -
```
faiss_doc_store.faiss_indexes[index].reset()
faiss_doc_store.faiss_indexes.pop(index)
faiss_doc_store.vector_dim = 384
# To update config (Optional step) need to implement `to_dict()` method
# faiss_doc_store.set_config(**faiss_doc_store.to_dict())
# Now call `update_embedding`
```

Resolving patch:
diff --git a/haystack/document_store/faiss.py b/haystack/document_store/faiss.py
--- a/haystack/document_store/faiss.py
+++ b/haystack/document_store/faiss.py
@@ -498,14 +498,14 @@ def load(cls, index_path: Union[str, Path], config_path: Optional[Union[str, Pat
         """
         if not config_path:
             index_path = Path(index_path)
-            faiss_init_params_path = index_path.with_suffix(".json")
+            config_path = index_path.with_suffix(".json")
 
         init_params: dict = {}
         try:
-            with open(faiss_init_params_path, 'r') as ipp:
+            with open(config_path, 'r') as ipp:
                 init_params = json.load(ipp)
         except OSError as e:
-            raise ValueError(f"Can't open FAISS configuration file `{faiss_init_params_path}`. "
+            raise ValueError(f"Can't open FAISS configuration file `{config_path}`. "
                              "Make sure the file exists and the you have the correct permissions "
                              "to access it.") from e
 


Test patch:
diff --git a/test/test_faiss_and_milvus.py b/test/test_faiss_and_milvus.py
--- a/test/test_faiss_and_milvus.py
+++ b/test/test_faiss_and_milvus.py
@@ -46,6 +46,34 @@ def test_faiss_index_save_and_load(tmp_path):
     assert not new_document_store.progress_bar
 
 
+def test_faiss_index_save_and_load_custom_path(tmp_path):
+    document_store = FAISSDocumentStore(
+        sql_url=f"sqlite:////{tmp_path/'haystack_test.db'}",
+        index="haystack_test",
+        progress_bar=False  # Just to check if the init parameters are kept
+    )
+    document_store.write_documents(DOCUMENTS)
+
+    # test saving the index
+    document_store.save(index_path=tmp_path / "haystack_test_faiss", config_path=tmp_path / "custom_path.json")
+
+    # clear existing faiss_index
+    document_store.faiss_indexes[document_store.index].reset()
+
+    # test faiss index is cleared
+    assert document_store.faiss_indexes[document_store.index].ntotal == 0
+
+    # test loading the index
+    new_document_store = FAISSDocumentStore.load(index_path=tmp_path / "haystack_test_faiss", config_path=tmp_path / "custom_path.json")
+
+    # check faiss index is restored
+    assert new_document_store.faiss_indexes[document_store.index].ntotal == len(DOCUMENTS)
+    # check if documents are restored
+    assert len(new_document_store.get_all_documents()) == len(DOCUMENTS)
+    # Check if the init parameters are kept
+    assert not new_document_store.progress_bar
+
+
 @pytest.mark.parametrize("document_store", ["faiss"], indirect=True)
 @pytest.mark.parametrize("index_buffer_size", [10_000, 2])
 @pytest.mark.parametrize("batch_size", [2])