Repository: deepset-ai/haystack

Issue:
OpenSearchDocumentStore: `No handler for type [flattened] declared on field [answer]`
### Discussed in https://github.com/deepset-ai/haystack/discussions/1603

<div type='discussions-op-text'>

<sup>Originally posted by **pallie97** October 15, 2021</sup>
Hello. I am trying to use AWS OpenSearch Service (version: OpenSearch1.0) for my document store -- I am using JupyterLab with Python 3.7 and the latest Haystack version. When I try to create the document store in my notebook with OpenSearchDocumentStore(), I receive the following error: 

`RequestError(400, 'mapper_parsing_exception', 'No handler for type [flattened] declared on field [answer]')`.


I am unsure if this is an AWS issue or if it's something the Haystack community can help me with. I have tried setting `index_type = 'hnsw' ` in the OpenSearchDocumentStore constructor and also tested different versions of ElasticSearch in AWS, but still get the same error. 

I appreciate any help, thanks! </div>


Discussion:
@tholor: probably related to #1398, do you know what might be happening?
I am able to reproduce the error and will investigate the root cause - very likely related to our recent refactoring in #1398 

Resolving patch:
diff --git a/haystack/document_store/elasticsearch.py b/haystack/document_store/elasticsearch.py
--- a/haystack/document_store/elasticsearch.py
+++ b/haystack/document_store/elasticsearch.py
@@ -1034,7 +1034,7 @@ def __init__(self,
                  scheme="https",
                  username="admin",
                  password="admin",
-                 port=9201,
+                 port=9200,
                  **kwargs):
 
         # Overwrite default kwarg values of parent class so that in default cases we can initialize
@@ -1200,6 +1200,36 @@ def _create_document_index(self, index_name: str):
             if not self.client.indices.exists(index=index_name):
                 raise e
 
+    def _create_label_index(self, index_name: str):
+        if self.client.indices.exists(index=index_name):
+            return
+        mapping = {
+            "mappings": {
+                "properties": {
+                    "query": {"type": "text"},
+                    "answer": {"type": "nested"}, # In elasticsearch we use type:flattened, but this is not supported in opensearch
+                    "document": {"type": "nested"},
+                    "is_correct_answer": {"type": "boolean"},
+                    "is_correct_document": {"type": "boolean"},
+                    "origin": {"type": "keyword"},  # e.g. user-feedback or gold-label
+                    "document_id": {"type": "keyword"},
+                    "no_answer": {"type": "boolean"},
+                    "pipeline_id": {"type": "keyword"},
+                    "created_at": {"type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"},
+                    "updated_at": {"type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"}
+                    #TODO add pipeline_hash and pipeline_name once we migrated the REST API to pipelines
+                }
+            }
+        }
+        try:
+            self.client.indices.create(index=index_name, body=mapping)
+        except RequestError as e:
+            # With multiple workers we need to avoid race conditions, where:
+            # - there's no index in the beginning
+            # - both want to create one
+            # - one fails as the other one already created it
+            if not self.client.indices.exists(index=index_name):
+                raise e
 
     def _get_vector_similarity_query(self, query_emb: np.ndarray, top_k: int):
         """


Test patch:
diff --git a/test/benchmarks/utils.py b/test/benchmarks/utils.py
--- a/test/benchmarks/utils.py
+++ b/test/benchmarks/utils.py
@@ -89,7 +89,7 @@ def get_document_store(document_store_type, similarity='dot_product', index="doc
             index_type = "flat"
         elif document_store_type == "opensearch_hnsw":
             index_type = "hnsw"
-        document_store = OpenSearchDocumentStore(index_type=index_type, timeout=3000)
+        document_store = OpenSearchDocumentStore(index_type=index_type, port=9201, timeout=3000)
     else:
         raise Exception(f"No document store fixture for '{document_store_type}'")
     return document_store