Repository: deepset-ai/haystack

Issue:
`write_labels` crashes for `ElasticsearchDocumentStore` for labels with long context
**Describe the bug**
`write_labels` currently crashes with a `BulkIndexError` if we try to write labels whose context is longer than 32766 bytes. I suspect that this is due to the mapping that we apply in `_create_label_index`. Probably we implicitly make use of type `keyword` instead of `text` for the `content` field of the `document` field of the label.

**Error message**
```
<class 'elasticsearch.helpers.errors.BulkIndexError'>, BulkIndexError('25 document(s) failed to index.', [{'index': {'_index': 'label', '_type': '_doc', '_id': '824075a5-9c6c-48af-b41c-d4e10d1d01d7', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'Document contains at least one immense term in field="document" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms...
```

**To Reproduce**
```python
from haystack.document_stores import ElasticsearchDocumentStore
from haystack.utils import fetch_archive_from_http, launch_es

# Download evaluation data
doc_dir = "data/"
s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/nq_dev_subset_v2.json.zip"
fetch_archive_from_http(url=s3_url, output_dir=doc_dir)

launch_es()
document_store = ElasticsearchDocumentStore()

document_store.add_eval_data(filename="data/nq_dev_subset_v2.json")
```



Discussion:
I'm happy to take a look at this tomorrow to see if I can debug this further
@bogdankostic Your intuition points in the right direction.

https://github.com/deepset-ai/haystack/blob/97a8d305129968dcbd5d483c849c1a778047f948/haystack/document_stores/elasticsearch.py#L225-L229

`document` is mapped as a [`flattened` field type](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/flattened.html#flattened):
> Given an object, the flattened mapping will parse out its leaf values and index them into one field as keywords.

Under the hood, document `content` is treated like a `keyword` and so it is not analyzed/split and is therefore considered a single immense term.

### Possible solutions
- use a regular `object` for `document` (and for `answer`, since there will be the same problem for this field).
It would be a simple and resolutive solution (I did some quick tests). But `flattened` was probably adopted in order to prevent performance issues (see [motivation for flattened data type](https://github.com/elastic/elasticsearch/issues/33003))...
- use the [`ignore_above` mapping parameter](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/ignore-above.html#ignore-above) to ignore strings longer than this limit (no indexing/storing). It doesn't seem like a good solution.

@masci @Winterflower any thoughts on this?
If we find together an acceptable solution, I can take charge of this issue.



### Related
In `OpensearchDocumentStore` we use `nested` instead of `flattened`, since `flattened` is not supported (see also https://github.com/deepset-ai/haystack/pull/1609):
https://github.com/deepset-ai/haystack/blob/21aedc644f701ea57463694c01fb980188ff0e45/haystack/document_stores/opensearch.py#L557-L564


**To solve the current bug, it may be reasonable to use [`nested`](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/nested.html)** also in `ElasticsearchDocumentStore`:
it is similar to `object`, but
> allows arrays of objects to be indexed in a way that they can be queried independently of each other.

WDYT?

Resolving patch:
diff --git a/haystack/document_stores/elasticsearch.py b/haystack/document_stores/elasticsearch.py
--- a/haystack/document_stores/elasticsearch.py
+++ b/haystack/document_stores/elasticsearch.py
@@ -521,8 +521,8 @@ def _create_label_index(self, index_name: str, headers: Optional[Dict[str, str]]
             "mappings": {
                 "properties": {
                     "query": {"type": "text"},
-                    "answer": {"type": "flattened"},  # light-weight but less search options than full object
-                    "document": {"type": "flattened"},
+                    "answer": {"type": "nested"},
+                    "document": {"type": "nested"},
                     "is_correct_answer": {"type": "boolean"},
                     "is_correct_document": {"type": "boolean"},
                     "origin": {"type": "keyword"},  # e.g. user-feedback or gold-label


Test patch:
diff --git a/test/document_stores/test_document_store.py b/test/document_stores/test_document_store.py
--- a/test/document_stores/test_document_store.py
+++ b/test/document_stores/test_document_store.py
@@ -877,6 +877,30 @@ def test_labels(document_store: BaseDocumentStore):
     assert len(labels) == 0
 
 
+@pytest.mark.parametrize("document_store", ["elasticsearch", "opensearch"], indirect=True)
+def test_labels_with_long_texts(document_store: BaseDocumentStore):
+    document_store.delete_index("label")
+    label = Label(
+        query="question1",
+        answer=Answer(
+            answer="answer",
+            type="extractive",
+            score=0.0,
+            context="something " * 10_000,
+            offsets_in_document=[Span(start=12, end=14)],
+            offsets_in_context=[Span(start=12, end=14)],
+        ),
+        is_correct_answer=True,
+        is_correct_document=True,
+        document=Document(content="something " * 10_000, id="123"),
+        origin="gold-label",
+    )
+    document_store.write_labels(labels=[label], index="label")
+    labels = document_store.get_all_labels(index="label")
+    assert len(labels) == 1
+    assert label == labels[0]
+
+
 # exclude weaviate because it does not support storing labels
 @pytest.mark.parametrize("document_store", ["elasticsearch", "faiss", "memory", "milvus1", "pinecone"], indirect=True)
 def test_multilabel(document_store: BaseDocumentStore):