Repository: deepset-ai/haystack
Pull request: Make `DeepsetCloudDocumentStore` work with non-existing index
**Proposed changes**:
This PR makes the `DeepsetCloudDocumentStore` work with a non-existing index. Non-existing index refers to either `None` (no index specified), or an index/pipeline that is not deployed on deepset Cloud.

**Status (please check what you already did)**:
- [x] First draft (up for discussions & feedback)
- [ ] Final code
- [x] Added tests
- [x] Updated documentation

Closes #2471


Problem statement:
Make `DeepsetCloudDocumentStore` work with non-existing index
We want to allow users to connect to the DC docstore without having to create a pipeline first. This will enable the worflow to create a pipeline from scratch using the DC SDK. 

Implementation: 
 - Set default index value to `None`
 - Make sure `info()` call does not raise an exception if index = `None` or specified index does not exist
 - always returns empty results for document stores that do not use an existing index
 - Logging of warnings such as `You are using a deepset cloud document store without an index. This document store will always return empty responses. To add files to your index, first upload files and publish&deploy your pipeline. DC will automatically index your files for you.`

Hints from issue discussion:

Commit message: Update Documentation & Code Style

Patch to review:
diff --git a/docs/_src/api/api/document_store.md b/docs/_src/api/api/document_store.md
index b816bd78f0..2cd829adc2 100644
--- a/docs/_src/api/api/document_store.md
+++ b/docs/_src/api/api/document_store.md
@@ -4014,7 +4014,7 @@ class DeepsetCloudDocumentStore(KeywordDocumentStore)
 #### \_\_init\_\_
 
 ```python
-def __init__(api_key: str = None, workspace: str = "default", index: str = "default", duplicate_documents: str = "overwrite", api_endpoint: Optional[str] = None, similarity: str = "dot_product", return_embedding: bool = False, label_index: str = "default")
+def __init__(api_key: str = None, workspace: str = "default", index: Optional[str] = None, duplicate_documents: str = "overwrite", api_endpoint: Optional[str] = None, similarity: str = "dot_product", return_embedding: bool = False, label_index: str = "default")
 ```
 
 A DocumentStore facade enabling you to interact with the documents stored in Deepset Cloud.
@@ -4030,8 +4030,9 @@ See https://haystack.deepset.ai/components/document-store for more information.
 If not specified, will be read from DEEPSET_CLOUD_API_KEY environment variable.
 See docs on how to generate an API key for your workspace: https://docs.cloud.deepset.ai/docs/connect-deepset-cloud-to-your-application
 - `workspace`: workspace name in Deepset Cloud
-- `index`: name of the index to access within the Deepset Cloud workspace. This equals typically the name of your pipeline.
-You can run Pipeline.list_pipelines_on_deepset_cloud() to see all available ones.
+- `index`: name of the index to access within the deepset Cloud workspace. This equals typically the name of
+your pipeline. You can run Pipeline.list_pipelines_on_deepset_cloud() to see all available ones.
+If you set index to `None`, this DocumentStore will always return empty results.
 - `duplicate_documents`: Handle duplicates document based on parameter options.
 Parameter options : ( 'skip','overwrite','fail')
 skip: Ignore the duplicates documents
diff --git a/haystack/document_stores/deepsetcloud.py b/haystack/document_stores/deepsetcloud.py
index c9dd6726f3..02cc1ef8cf 100644
--- a/haystack/document_stores/deepsetcloud.py
+++ b/haystack/document_stores/deepsetcloud.py
@@ -17,7 +17,7 @@ def __init__(
         self,
         api_key: str = None,
         workspace: str = "default",
-        index: str = "default",
+        index: Optional[str] = None,
         duplicate_documents: str = "overwrite",
         api_endpoint: Optional[str] = None,
         similarity: str = "dot_product",
@@ -35,8 +35,9 @@ def __init__(
                         If not specified, will be read from DEEPSET_CLOUD_API_KEY environment variable.
                         See docs on how to generate an API key for your workspace: https://docs.cloud.deepset.ai/docs/connect-deepset-cloud-to-your-application
         :param workspace: workspace name in Deepset Cloud
-        :param index: name of the index to access within the Deepset Cloud workspace. This equals typically the name of your pipeline.
-                      You can run Pipeline.list_pipelines_on_deepset_cloud() to see all available ones.
+        :param index: name of the index to access within the deepset Cloud workspace. This equals typically the name of
+                      your pipeline. You can run Pipeline.list_pipelines_on_deepset_cloud() to see all available ones.
+                      If you set index to `None`, this DocumentStore will always return empty results.
         :param duplicate_documents: Handle duplicates document based on parameter options.
                                     Parameter options : ( 'skip','overwrite','fail')
                                     skip: Ignore the duplicates documents
@@ -61,12 +62,32 @@ def __init__(
         self.client = DeepsetCloud.get_index_client(
             api_key=api_key, api_endpoint=api_endpoint, workspace=workspace, index=index
         )
+        # Check if index exists
+        pipeline_client = DeepsetCloud.get_pipeline_client(
+            api_key=api_key, api_endpoint=api_endpoint, workspace=workspace
+        )
 
-        index_info = self.client.info()
-        indexing_info = index_info["indexing"]
-        if indexing_info["pending_file_count"] > 0:
+        deployed_pipelines = set(
+            pipe["name"]
+            for pipe in pipeline_client.list_pipeline_configs(workspace=workspace)
+            if pipe["status"] == "DEPLOYED"
+        )
+        self.index_exists = index in deployed_pipelines
+
+        if self.index_exists:
+            index_info = self.client.info()
+            indexing_info = index_info["indexing"]
+            if indexing_info["pending_file_count"] > 0:
+                logger.warning(
+                    f"{indexing_info['pending_file_count']} files are pending to be indexed. "
+                    f"Indexing status: {indexing_info['status']}"
+                )
+        else:
             logger.warning(
-                f"{indexing_info['pending_file_count']} files are pending to be indexed. Indexing status: {indexing_info['status']}"
+                f"You are using a DeepsetCloudDocumentStore with an index that does not exist on deepset Cloud. "
+                f"This document store will always return empty responses. To add files to your index, first upload "
+                f"files and publish&deploy your pipeline. "
+                f"DC will automatically index your files for you."
             )
 
         self.evaluation_set_client = DeepsetCloud.get_evaluation_set_client(
@@ -122,6 +143,9 @@ def get_all_documents(
             "`get_all_documents()` can get very slow and resource-heavy since all documents must be loaded from Deepset Cloud. "
             "Consider using `get_all_documents_generator()` instead."
         )
+        if not self.index_exists:
+            return []
+
         return list(
             self.get_all_documents_generator(
                 index=index, filters=filters, return_embedding=return_embedding, batch_size=batch_size, headers=headers
@@ -173,6 +197,9 @@ def get_all_documents_generator(
         :param batch_size: When working with large number of documents, batching can help reduce memory footprint.
         :param headers: Custom HTTP headers to pass to document store client if supported (e.g. {'Authorization': 'Basic YWRtaW46cm9vdA=='} for basic authentication)
         """
+        if not self.index_exists:
+            return
+
         if batch_size != 10_000:
             raise ValueError("DeepsetCloudDocumentStore does not support batching")
 
@@ -192,6 +219,9 @@ def get_all_documents_generator(
     def get_document_by_id(
         self, id: str, index: Optional[str] = None, headers: Optional[Dict[str, str]] = None
     ) -> Optional[Document]:
+        if not self.index_exists:
+            return None
+
         if index is None:
             index = self.index
 
@@ -209,6 +239,9 @@ def get_documents_by_id(
         batch_size: int = 10_000,
         headers: Optional[Dict[str, str]] = None,
     ) -> List[Document]:
+        if not self.index_exists:
+            return []
+
         if batch_size != 10_000:
             raise ValueError("DeepsetCloudDocumentStore does not support batching")
 
@@ -222,6 +255,9 @@ def get_document_count(
         only_documents_without_embedding: bool = False,
         headers: Optional[Dict[str, str]] = None,
     ) -> int:
+        if not self.index_exists:
+            return 0
+
         count_result = self.client.count_documents(
             filters=filters,
             only_documents_without_embedding=only_documents_without_embedding,
@@ -316,6 +352,9 @@ def query_by_embedding(
                             Otherwise raw similarity scores (e.g. cosine or dot_product) will be used.
         :return:
         """
+        if not self.index_exists:
+            return []
+
         if return_embedding is None:
             return_embedding = self.return_embedding
 
@@ -423,6 +462,9 @@ def query(
                             If true (default) similarity scores (e.g. cosine or dot_product) which naturally have a different value range will be scaled to a range of [0,1], where 1 means extremely relevant.
                             Otherwise raw similarity scores (e.g. cosine or dot_product) will be used.
         """
+        if not self.index_exists:
+            return []
+
         doc_dicts = self.client.query(
             query=query,
             filters=filters,
diff --git a/haystack/json-schemas/haystack-pipeline-master.schema.json b/haystack/json-schemas/haystack-pipeline-master.schema.json
index af54e3db0e..a598258604 100644
--- a/haystack/json-schemas/haystack-pipeline-master.schema.json
+++ b/haystack/json-schemas/haystack-pipeline-master.schema.json
@@ -268,7 +268,6 @@
             },
             "index": {
               "title": "Index",
-              "default": "default",
               "type": "string"
             },
             "duplicate_documents": {
diff --git a/test/conftest.py b/test/conftest.py
index f7bf2b5c7a..303e3659b8 100644
--- a/test/conftest.py
+++ b/test/conftest.py
@@ -399,6 +399,22 @@ def deepset_cloud_fixture():
             json={"indexing": {"status": "INDEXED", "pending_file_count": 0, "total_file_count": 31}},
             status=200,
         )
+        responses.add(
+            method=responses.GET,
+            url=f"{DC_API_ENDPOINT}/workspaces/default/pipelines",
+            match=[responses.matchers.header_matcher({"authorization": f"Bearer {DC_API_KEY}"})],
+            json={
+                "data": [
+                    {
+                        "name": DC_TEST_INDEX,
+                        "status": "DEPLOYED",
+                        "indexing": {"status": "INDEXED", "pending_file_count": 0, "total_file_count": 31},
+                    }
+                ],
+                "has_more": False,
+                "total": 1,
+            },
+        )
     else:
         responses.add_passthru(DC_API_ENDPOINT)
 
diff --git a/test/test_document_store.py b/test/test_document_store.py
index 9695dda6d4..26f38d55c3 100644
--- a/test/test_document_store.py
+++ b/test/test_document_store.py
@@ -1883,6 +1883,56 @@ def test_DeepsetCloudDocumentStore_query_by_embedding(deepset_cloud_document_sto
     assert len(emb_docs) == 0
 
 
+@pytest.mark.usefixtures(deepset_cloud_fixture.__name__)
+@responses.activate
+def test_DeepsetCloudDocumentStore_get_all_docs_without_index():
+    document_store = DeepsetCloudDocumentStore(api_endpoint=DC_API_ENDPOINT, api_key=DC_API_KEY, index=None)
+    assert document_store.get_all_documents() == []
+
+
+@pytest.mark.usefixtures(deepset_cloud_fixture.__name__)
+@responses.activate
+def test_DeepsetCloudDocumentStore_get_all_docs_generator_without_index():
+    document_store = DeepsetCloudDocumentStore(api_endpoint=DC_API_ENDPOINT, api_key=DC_API_KEY, index=None)
+    assert list(document_store.get_all_documents_generator()) == []
+
+
+@pytest.mark.usefixtures(deepset_cloud_fixture.__name__)
+@responses.activate
+def test_DeepsetCloudDocumentStore_get_doc_by_id_without_index():
+    document_store = DeepsetCloudDocumentStore(api_endpoint=DC_API_ENDPOINT, api_key=DC_API_KEY, index=None)
+    assert document_store.get_document_by_id(id="some id") == None
+
+
+@pytest.mark.usefixtures(deepset_cloud_fixture.__name__)
+@responses.activate
+def test_DeepsetCloudDocumentStore_get_docs_by_id_without_index():
+    document_store = DeepsetCloudDocumentStore(api_endpoint=DC_API_ENDPOINT, api_key=DC_API_KEY, index=None)
+    assert document_store.get_documents_by_id(ids=["some id"]) == []
+
+
+@pytest.mark.usefixtures(deepset_cloud_fixture.__name__)
+@responses.activate
+def test_DeepsetCloudDocumentStore_get_doc_count_without_index():
+    document_store = DeepsetCloudDocumentStore(api_endpoint=DC_API_ENDPOINT, api_key=DC_API_KEY, index=None)
+    assert document_store.get_document_count() == 0
+
+
+@pytest.mark.usefixtures(deepset_cloud_fixture.__name__)
+@responses.activate
+def test_DeepsetCloudDocumentStore_query_by_emb_without_index():
+    query_emb = np.random.randn(768)
+    document_store = DeepsetCloudDocumentStore(api_endpoint=DC_API_ENDPOINT, api_key=DC_API_KEY, index=None)
+    assert document_store.query_by_embedding(query_emb=query_emb) == []
+
+
+@pytest.mark.usefixtures(deepset_cloud_fixture.__name__)
+@responses.activate
+def test_DeepsetCloudDocumentStore_query_without_index():
+    document_store = DeepsetCloudDocumentStore(api_endpoint=DC_API_ENDPOINT, api_key=DC_API_KEY, index=None)
+    assert document_store.query(query="some query") == []
+
+
 @pytest.mark.elasticsearch
 def test_elasticsearch_search_field_mapping():
 


Code review comments:
diff_hunk: @@ -61,12 +62,32 @@ def __init__(
         self.client = DeepsetCloud.get_index_client(
             api_key=api_key, api_endpoint=api_endpoint, workspace=workspace, index=index
         )
+        # Check if index exists
+        pipeline_client = DeepsetCloud.get_pipeline_client(
+            api_key=api_key, api_endpoint=api_endpoint, workspace=workspace
+        )
 
-        index_info = self.client.info()
-        indexing_info = index_info["indexing"]
-        if indexing_info["pending_file_count"] > 0:
+        deployed_pipelines = set(
+            pipe["name"]
+            for pipe in pipeline_client.list_pipeline_configs(workspace=workspace)
+            if pipe["status"] == "DEPLOYED"
path: haystack/document_stores/deepsetcloud.py
text: @user1:
There's another state "DEPLOYED_UNHEALTHY" where the index exists too.
This might happen if there was an issue during indexing. The index itself is queriable however. So let's add it here and additionally show a warning that the index is unhealthy and should be redeployed using `Pipeline.undeploy_on_deepset_cloud()` and `Pipeline.deploy_on_deepset_cloud()`.

@author:
Done.

diff_hunk: @@ -61,12 +62,32 @@ def __init__(
         self.client = DeepsetCloud.get_index_client(
             api_key=api_key, api_endpoint=api_endpoint, workspace=workspace, index=index
         )
+        # Check if index exists
+        pipeline_client = DeepsetCloud.get_pipeline_client(
+            api_key=api_key, api_endpoint=api_endpoint, workspace=workspace
+        )
 
-        index_info = self.client.info()
-        indexing_info = index_info["indexing"]
-        if indexing_info["pending_file_count"] > 0:
+        deployed_pipelines = set(
+            pipe["name"]
+            for pipe in pipeline_client.list_pipeline_configs(workspace=workspace)
+            if pipe["status"] == "DEPLOYED"
+        )
+        self.index_exists = index in deployed_pipelines
+
+        if self.index_exists:
+            index_info = self.client.info()
+            indexing_info = index_info["indexing"]
+            if indexing_info["pending_file_count"] > 0:
+                logger.warning(
+                    f"{indexing_info['pending_file_count']} files are pending to be indexed. "
+                    f"Indexing status: {indexing_info['status']}"
+                )
+        else:
             logger.warning(
-                f"{indexing_info['pending_file_count']} files are pending to be indexed. Indexing status: {indexing_info['status']}"
+                f"You are using a DeepsetCloudDocumentStore with an index that does not exist on deepset Cloud. "
+                f"This document store will always return empty responses. To add files to your index, first upload "
+                f"files and publish&deploy your pipeline. "
+                f"DC will automatically index your files for you."
path: haystack/document_stores/deepsetcloud.py
text: @user1:
I would convert that into an info as this is now a common case. Additionally I'd rather add a hint on how to create the index on deepset cloud: 
```suggestion
                f"This document store will always return empty responses. This is especially useful if you want to create a new pipeline within deepset Cloud.\n"
                f"In order to create a new pipeline on deepset Cloud, take the following steps: \n"
                f"  - create query and index pipelines using this DocumentStore\n"
                f"  - call `Pipeline.save_to_deepset_cloud()` passing the pipelines and a `pipeline_config_name`\n"
                f"  - call `Pipeline.deploy_on_deepset_cloud()` passing the `pipeline_config_name`"
```

@author:
Done.