Repository: deepset-ai/haystack

Issue:
OpenSearchDocumentStore: default username leads to not using AWS4Auth, which leads to an error
**Describe the bug**

When you want to use AWS4Auth to connect to the OpenSearchDocumentStore, then you need to set `username` to `None`, otherwise the OpenSearchDocumentStore will be initialised without AWS4Auth and you will get an error.  This is not obvious to a user, therefore I would suggest to remove the default `"admin"` for the `username` and `password` argument.

I would submit a PR for that. But, is that set on purpose to make it easy to connect to a default OpenSearch in e.g. Docker? Should instead of removing the default arguments the `elif aws4auth` block be evaluated first?

The issue is, that here, default values are set

https://github.com/deepset-ai/haystack/blob/77cea8b1408699f69dc09420f9f9af3cf5e6f47a/haystack/document_stores/opensearch.py#L38-L47

And then here the first if block is executed, instead of the desired `aws4auth` block.

https://github.com/deepset-ai/haystack/blob/77cea8b1408699f69dc09420f9f9af3cf5e6f47a/haystack/document_stores/opensearch.py#L236-L266

**Error message**
The partial stack trace is then:

```
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    connect_aws_opensearch_document_store("xxx.eu-central-1.es.amazonaws.com", 443, "eu-central-1")
  File "test.py", line 13, in connect_aws_opensearch_document_store
    document_store = OpenSearchDocumentStore(host=host_address, aws4auth=auth,ca_certs=certifi.where(), port=port)
  File "/home/ec2-user/opensearch_test/.venv/lib64/python3.7/site-packages/haystack/nodes/base.py", line 48, in wrapper_exportable_to_yaml
    init_func(self, *args, **kwargs)
  File "/home/ec2-user/opensearch_test/.venv/lib64/python3.7/site-packages/haystack/document_stores/opensearch.py", line 177, in __init__
    f"Initial connection to Opensearch failed with error '{e}'\n"
ConnectionError: Initial connection to Opensearch failed with error 'AuthorizationException(403, '{"message":"Authorization header requires \'Credential\' parameter. Authorization header requires \'Signature\' parameter. Authorization header requires \'SignedHeaders\' parameter. Authorization header requires existence of either a \'X-Amz-Date\' or a \'Date\' header. Authorization=Basic YWRtaW46YWRtaW4="}')'
Make sure an Opensearch instance is running at `xxx.eu-central-1.es.amazonaws.com` and that it has finished booting (can take > 30s).
```

**Expected behavior**

If I use `OpenSearchDocumentStore(host=host_address, aws4auth=auth,ca_certs=certifi.where(), port=port)` then it connects using `AWS4Auth`.


**To Reproduce**
On an EC2 instance with the OpenSearchFullAccess policy attached to its role:

```python
import boto3
import certifi
from requests_aws4auth import AWS4Auth
from haystack.document_stores import OpenSearchDocumentStore

host_address="insert_adress_here"
port=443
region="eu-central-1"

credentials = boto3.Session().get_credentials()
access_key = credentials.access_key
secret_key = credentials.secret_key
service = 'es'
session_token = credentials.token
auth = AWS4Auth(access_key, secret_key, region, service, session_token=session_token)

document_store = OpenSearchDocumentStore(host=host_address, aws4auth=auth,ca_certs=certifi.where(), port=port) # errors
document_store = OpenSearchDocumentStore(username=None, host=host_address, aws4auth=auth,ca_certs=certifi.where(), port=port) # works
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:
@FabianHertwig Thank you for bringing this up. It seems to me that we should improve documentation here. Your assumption is correct that we have the username and password set to `admin` by default because it's the default in the opensearch docker containers: https://opensearch.org/docs/1.3/opensearch/install/docker/#run-opensearch-in-a-docker-container
If we change the default value in Haystack's code to `None`, it would be a breaking change for many users experimenting with OpenSearch docker containers and it might cause a lot of trouble for them. (It shouldn't affect production use cases with this username and password obviously.)
Changing the order of the `if` and `elif` branch would work for them and also for your use case. If you would like to do that in a PR and contribute to Haystack you have my support! 🙂  
What the PR could do also do is to add an `if username and aws4auth` check in the very beginning and show an error message in that case. The error message could explain that only either `username` or `aws4auth` are allowed to be set.
In my opinion, at least there should be a warning if the first `if` branch is executed although another condition is also true here.

Resolving patch:
diff --git a/haystack/document_stores/elasticsearch.py b/haystack/document_stores/elasticsearch.py
--- a/haystack/document_stores/elasticsearch.py
+++ b/haystack/document_stores/elasticsearch.py
@@ -231,6 +231,10 @@ def _init_elastic_client(
         elif aws4auth:
             # aws elasticsearch with IAM
             # see https://elasticsearch-py.readthedocs.io/en/v7.12.0/index.html?highlight=http_auth#running-on-aws-with-iam
+            if username:
+                logger.warning(
+                    "aws4auth and a username are passed to the ElasticsearchDocumentStore. The username will be ignored and aws4auth will be used for authentication."
+                )
             client = Elasticsearch(
                 hosts=hosts,
                 http_auth=aws4auth,
diff --git a/haystack/document_stores/opensearch.py b/haystack/document_stores/opensearch.py
--- a/haystack/document_stores/opensearch.py
+++ b/haystack/document_stores/opensearch.py
@@ -238,20 +238,13 @@ def _init_client(
         if use_system_proxy:
             connection_class = RequestsHttpConnection  # type: ignore [assignment]
 
-        if username:
-            # standard http_auth
-            client = OpenSearch(
-                hosts=hosts,
-                http_auth=(username, password),
-                scheme=scheme,
-                ca_certs=ca_certs,
-                verify_certs=verify_certs,
-                timeout=timeout,
-                connection_class=connection_class,
-            )
-        elif aws4auth:
+        if aws4auth:
             # Sign requests to Opensearch with IAM credentials
             # see https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-python
+            if username:
+                logger.warning(
+                    "aws4auth and a username or the default username 'admin' are passed to the OpenSearchDocumentStore. The username will be ignored and aws4auth will be used for authentication."
+                )
             client = OpenSearch(
                 hosts=hosts,
                 http_auth=aws4auth,
@@ -260,6 +253,17 @@ def _init_client(
                 verify_certs=True,
                 timeout=timeout,
             )
+        elif username:
+            # standard http_auth
+            client = OpenSearch(
+                hosts=hosts,
+                http_auth=(username, password),
+                scheme=scheme,
+                ca_certs=ca_certs,
+                verify_certs=verify_certs,
+                timeout=timeout,
+                connection_class=connection_class,
+            )
         else:
             # no authentication needed
             client = OpenSearch(


Test patch:
diff --git a/test/document_stores/test_elasticsearch.py b/test/document_stores/test_elasticsearch.py
--- a/test/document_stores/test_elasticsearch.py
+++ b/test/document_stores/test_elasticsearch.py
@@ -1,11 +1,12 @@
+import logging
 import os
 from unittest.mock import MagicMock
-import pytest
 
 import numpy as np
+import pytest
 
+from haystack.document_stores.elasticsearch import ElasticsearchDocumentStore, Elasticsearch
 from haystack.schema import Document
-from haystack.document_stores.elasticsearch import ElasticsearchDocumentStore
 
 from .test_base import DocumentStoreBaseTestAbstract
 from .test_search_engine import SearchEngineDocumentStoreTestAbstract
@@ -32,6 +33,18 @@ def ds(self):
         ds.delete_index(self.index_name)
         ds.delete_index(labels_index_name)
 
+    @pytest.fixture
+    def mocked_elastic_search_init(self, monkeypatch):
+        mocked_init = MagicMock(return_value=None)
+        monkeypatch.setattr(Elasticsearch, "__init__", mocked_init)
+        return mocked_init
+
+    @pytest.fixture
+    def mocked_elastic_search_ping(self, monkeypatch):
+        mocked_ping = MagicMock(return_value=True)
+        monkeypatch.setattr(Elasticsearch, "ping", mocked_ping)
+        return mocked_ping
+
     @pytest.fixture
     def mocked_document_store(self):
         """
@@ -239,3 +252,38 @@ def test_get_document_count_only_documents_without_embedding_arg(self, ds, docum
         assert ds.get_document_count(only_documents_without_embedding=True) == 3
         assert ds.get_document_count(only_documents_without_embedding=True, filters={"month": ["01"]}) == 0
         assert ds.get_document_count(only_documents_without_embedding=True, filters={"month": ["03"]}) == 3
+
+    @pytest.mark.unit
+    def test__init_elastic_client_aws4auth_and_username_raises_warning(
+        self, caplog, mocked_elastic_search_init, mocked_elastic_search_ping
+    ):
+        _init_client_remaining_kwargs = {
+            "host": "host",
+            "port": 443,
+            "password": "pass",
+            "api_key_id": None,
+            "api_key": None,
+            "scheme": "https",
+            "ca_certs": None,
+            "verify_certs": True,
+            "timeout": 10,
+            "use_system_proxy": False,
+        }
+
+        with caplog.at_level(logging.WARN, logger="haystack.document_stores.elasticsearch"):
+            ElasticsearchDocumentStore._init_elastic_client(
+                username="admin", aws4auth="foo", **_init_client_remaining_kwargs
+            )
+        assert len(caplog.records) == 1
+        for r in caplog.records:
+            assert r.levelname == "WARNING"
+
+        caplog.clear()
+        with caplog.at_level(logging.WARN, logger="haystack.document_stores.elasticsearch"):
+            ElasticsearchDocumentStore._init_elastic_client(
+                username=None, aws4auth="foo", **_init_client_remaining_kwargs
+            )
+            ElasticsearchDocumentStore._init_elastic_client(
+                username="", aws4auth="foo", **_init_client_remaining_kwargs
+            )
+        assert len(caplog.records) == 0
diff --git a/test/document_stores/test_opensearch.py b/test/document_stores/test_opensearch.py
--- a/test/document_stores/test_opensearch.py
+++ b/test/document_stores/test_opensearch.py
@@ -199,6 +199,32 @@ def test___init___api_key_raises_warning(self, mocked_document_store, caplog):
         for r in caplog.records:
             assert r.levelname == "WARNING"
 
+    @pytest.mark.unit
+    def test__init_client_aws4auth_and_username_raises_warning(self, mocked_open_search_init, caplog):
+        _init_client_remaining_kwargs = {
+            "host": "host",
+            "port": 443,
+            "password": "pass",
+            "scheme": "https",
+            "ca_certs": None,
+            "verify_certs": True,
+            "timeout": 10,
+            "use_system_proxy": False,
+        }
+
+        with caplog.at_level(logging.WARN, logger="haystack.document_stores.opensearch"):
+            OpenSearchDocumentStore._init_client(username="admin", aws4auth="foo", **_init_client_remaining_kwargs)
+            OpenSearchDocumentStore._init_client(username="bar", aws4auth="foo", **_init_client_remaining_kwargs)
+        assert len(caplog.records) == 2
+        for r in caplog.records:
+            assert r.levelname == "WARNING"
+
+        caplog.clear()
+        with caplog.at_level(logging.WARN, logger="haystack.document_stores.opensearch"):
+            OpenSearchDocumentStore._init_client(username=None, aws4auth="foo", **_init_client_remaining_kwargs)
+            OpenSearchDocumentStore._init_client(username="foo", aws4auth=None, **_init_client_remaining_kwargs)
+        assert len(caplog.records) == 0
+
     @pytest.mark.unit
     def test___init___connection_test_fails(self, mocked_document_store):
         failing_client = MagicMock()