Repository: apache/airflow

Issue:
Migrate elasticsearch client to 8.* (or latest 7.*)
### Body

The eleasticsearch version we use is old (from 2021). It is eleasticsearch 7.* and since we are using elasticsearch-dsl, we cannot used elasticsearch 8 currently (first version of that was also released in 2021). 

We are currently limited to < 7.15.0 additionally to being limited to 7, because 7.15.0 breaks our tests as we found out when we released some of the dependencies as part of https://github.com/apache/airflow/pull/30067

We should migrate to a newer version of elasticsearch. There are two options:

1) Fix the tests and migrate to lates 7.* version - and bump to latest released 7* by fixing our tests (possible for short term).
2) Migreate to 8 version (better long term)

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


Discussion:
Since `elasticsearch-dsl` doesn't support elasticsearch 8, if we are to migrate to the official elasticsearch 8, does it mean we need to deprecate `elasticsearch-dsl` also ?
If we do yes. But that also might be reason why we don't do it.
Ok. Please assign this issue to me. I'll take some time to study the current usage of this package and propose whether to upgrade to 8.* or latest 7.*
Done.
I saw `elasticsearch_dsl` was only used in 3 files as follows (one usage in taskhandler & 2 in testing) : 

https://github.com/apache/airflow/blob/fc36d002fccaad65718f42994747600c14f10263/airflow/providers/elasticsearch/log/es_task_handler.py#L291
https://github.com/apache/airflow/blob/fc36d002fccaad65718f42994747600c14f10263/docker_tests/test_prod_image.py#L130
https://github.com/apache/airflow/blob/fc36d002fccaad65718f42994747600c14f10263/tests/providers/elasticsearch/log/test_es_task_handler.py#L344

Given the utilization of `elasticsearch_dsl` is fairly limited, I'll try to upgrade to `elasticsearch` 8.* and deprecate `elasticsearch_dsl` 
hmmmm - I just realize that we also use `elasticsearch-dbapi`, which requires `elasticsearch==7.13.4`

https://github.com/apache/airflow/blob/main/airflow/providers/elasticsearch/provider.yaml#L60
https://github.com/preset-io/elasticsearch-dbapi/blob/master/requirements.txt#L9
I feel like if we are to upgrade to elasticsearch 8, we need to divide it into chunk of works (coz we will be deprecating 2 packages). Since `elasticsearch-dbapi` is used to construct the `ElasticsearchSQLHook`, we might first need to refactor it using the existing official `elasticsearch` library. Then we can go further to refactor the taskhandler that uses `elasticsearch_dsl` to use the existing `elasticsearch` library. Finally we can attempt to upgrade to 8, remove those 2 libraries, and fix the failing tests. What do you think ? @potiuk   
sure

Resolving patch:
diff --git a/airflow/providers/elasticsearch/log/es_task_handler.py b/airflow/providers/elasticsearch/log/es_task_handler.py
--- a/airflow/providers/elasticsearch/log/es_task_handler.py
+++ b/airflow/providers/elasticsearch/log/es_task_handler.py
@@ -30,7 +30,7 @@
 # Using `from elasticsearch import *` would break elasticsearch mocking used in unit test.
 import elasticsearch
 import pendulum
-from elasticsearch.exceptions import ElasticsearchException, NotFoundError
+from elasticsearch.exceptions import NotFoundError
 
 from airflow.configuration import conf
 from airflow.exceptions import AirflowProviderDeprecationWarning
@@ -89,7 +89,7 @@ def __init__(
         json_fields: str,
         host_field: str = "host",
         offset_field: str = "offset",
-        host: str = "localhost:9200",
+        host: str = "http://localhost:9200",
         frontend: str = "localhost:5601",
         index_patterns: str | None = conf.get("elasticsearch", "index_patterns", fallback="_all"),
         es_kwargs: dict | None = conf.getsection("elasticsearch_configs"),
@@ -101,8 +101,8 @@ def __init__(
         super().__init__(base_log_folder, filename_template)
         self.closed = False
 
-        self.client = elasticsearch.Elasticsearch(host.split(";"), **es_kwargs)  # type: ignore[attr-defined]
-
+        self.client = elasticsearch.Elasticsearch(host, **es_kwargs)  # type: ignore[attr-defined]
+        # in airflow.cfg, host of elasticsearch has to be http://dockerhostXxxx:9200
         if USE_PER_RUN_LOG_ID and log_id_template is not None:
             warnings.warn(
                 "Passing log_id_template to ElasticsearchTaskHandler is deprecated and has no effect",
@@ -292,27 +292,24 @@ def es_read(self, log_id: str, offset: int | str, metadata: dict) -> list | Elas
         }
 
         try:
-            max_log_line = self.client.count(index=self.index_patterns, body=query)["count"]
+            max_log_line = self.client.count(index=self.index_patterns, body=query)["count"]  # type: ignore
         except NotFoundError as e:
             self.log.exception("The target index pattern %s does not exist", self.index_patterns)
             raise e
-        except ElasticsearchException as e:
-            self.log.exception("Could not get current log size with log_id: %s", log_id)
-            raise e
 
         logs: list[Any] | ElasticSearchResponse = []
         if max_log_line != 0:
             try:
                 query.update({"sort": [self.offset_field]})
-                res = self.client.search(
+                res = self.client.search(  # type: ignore
                     index=self.index_patterns,
                     body=query,
                     size=self.MAX_LINE_PER_PAGE,
                     from_=self.MAX_LINE_PER_PAGE * self.PAGE,
                 )
                 logs = ElasticSearchResponse(self, res)
-            except elasticsearch.exceptions.ElasticsearchException:
-                self.log.exception("Could not read log with log_id: %s", log_id)
+            except Exception as err:
+                self.log.exception("Could not read log with log_id: %s. Exception: %s", log_id, err)
 
         return logs
 


Test patch:
diff --git a/tests/providers/elasticsearch/log/elasticmock/__init__.py b/tests/providers/elasticsearch/log/elasticmock/__init__.py
--- a/tests/providers/elasticsearch/log/elasticmock/__init__.py
+++ b/tests/providers/elasticsearch/log/elasticmock/__init__.py
@@ -41,17 +41,55 @@
 """Elastic mock module used for testing"""
 from functools import wraps
 from unittest.mock import patch
-
-from elasticsearch.client.utils import _normalize_hosts
+from urllib.parse import unquote, urlparse
 
 from .fake_elasticsearch import FakeElasticsearch
 
 ELASTIC_INSTANCES: dict[str, FakeElasticsearch] = {}
 
 
+def _normalize_hosts(hosts):
+    """
+    Helper function to transform hosts argument to
+    :class:`~elasticsearch.Elasticsearch` to a list of dicts.
+    """
+    # if hosts are empty, just defer to defaults down the line
+    if hosts is None:
+        return [{}]
+
+    hosts = [hosts]
+
+    out = []
+
+    for host in hosts:
+        if "://" not in host:
+            host = f"//{host}"
+
+        parsed_url = urlparse(host)
+        h = {"host": parsed_url.hostname}
+
+        if parsed_url.port:
+            h["port"] = parsed_url.port
+
+        if parsed_url.scheme == "https":
+            h["port"] = parsed_url.port or 443
+            h["use_ssl"] = True
+
+        if parsed_url.username or parsed_url.password:
+            h["http_auth"] = f"{unquote(parsed_url.username)}:{unquote(parsed_url.password)}"
+
+        if parsed_url.path and parsed_url.path != "/":
+            h["url_prefix"] = parsed_url.path
+
+        out.append(h)
+    else:
+        out.append(host)
+    return out
+
+
 def _get_elasticmock(hosts=None, *args, **kwargs):
     host = _normalize_hosts(hosts)[0]
-    elastic_key = f"{host.get('host', 'localhost')}:{host.get('port', 9200)}"
+    elastic_key = f"http://{host.get('host', 'localhost')}:{host.get('port', 9200)}"
 
     if elastic_key in ELASTIC_INSTANCES:
         connection = ELASTIC_INSTANCES.get(elastic_key)
diff --git a/tests/providers/elasticsearch/log/elasticmock/fake_elasticsearch.py b/tests/providers/elasticsearch/log/elasticmock/fake_elasticsearch.py
--- a/tests/providers/elasticsearch/log/elasticmock/fake_elasticsearch.py
+++ b/tests/providers/elasticsearch/log/elasticmock/fake_elasticsearch.py
@@ -20,10 +20,9 @@
 import json
 
 from elasticsearch import Elasticsearch
-from elasticsearch.client.utils import query_params
 from elasticsearch.exceptions import NotFoundError
 
-from .utilities import get_random_id
+from .utilities import MissingIndexException, get_random_id, query_params
 
 #
 # The MIT License (MIT)
@@ -53,7 +52,7 @@ class FakeElasticsearch(Elasticsearch):
     __documents_dict = None
 
     def __init__(self):
-        super().__init__()
+        super().__init__("http://localhost:9200")
         self.__documents_dict = {}
 
     @query_params()
@@ -327,9 +326,8 @@ def get_source(self, index, doc_type, id, params=None):
         "version",
     )
     def count(self, index=None, doc_type=None, body=None, params=None, headers=None):
-        searchable_indexes = self._normalize_index_to_list(index)
+        searchable_indexes = self._normalize_index_to_list(index, body)
         searchable_doc_types = self._normalize_doc_type_to_list(doc_type)
-
         i = 0
         for searchable_index in searchable_indexes:
             for document in self.__documents_dict[searchable_index]:
@@ -376,7 +374,7 @@ def count(self, index=None, doc_type=None, body=None, params=None, headers=None)
         "version",
     )
     def search(self, index=None, doc_type=None, body=None, params=None, headers=None):
-        searchable_indexes = self._normalize_index_to_list(index)
+        searchable_indexes = self._normalize_index_to_list(index, body)
 
         matches = self._find_match(index, doc_type, body)
 
@@ -446,7 +444,7 @@ def suggest(self, body, index=None):
         return result_dict
 
     def _find_match(self, index, doc_type, body):
-        searchable_indexes = self._normalize_index_to_list(index)
+        searchable_indexes = self._normalize_index_to_list(index, body)
         searchable_doc_types = self._normalize_doc_type_to_list(doc_type)
 
         must = body["query"]["bool"]["must"][0]  # only support one must
@@ -477,19 +475,20 @@ def match_must_phrase(document, matches, must):
                     matches.append(document)
 
     # Check index(es) exists.
-    def _validate_search_targets(self, targets):
+    def _validate_search_targets(self, targets, body):
         # TODO: support allow_no_indices query parameter
         matches = set()
         for target in targets:
+            print(f"Loop over:::target = {target}")
             if target == "_all" or target == "":
                 matches.update(self.__documents_dict)
             elif "*" in target:
                 matches.update(fnmatch.filter(self.__documents_dict, target))
             elif target not in self.__documents_dict:
-                raise NotFoundError(404, f"IndexMissingException[[{target}] missing]")
+                raise MissingIndexException(msg=f"IndexMissingException[[{target}] missing]", body=body)
         return matches
 
-    def _normalize_index_to_list(self, index):
+    def _normalize_index_to_list(self, index, body):
         # Ensure to have a list of index
         if index is None:
             searchable_indexes = self.__documents_dict.keys()
@@ -501,11 +500,8 @@ def _normalize_index_to_list(self, index):
             # Is it the correct exception to use ?
             raise ValueError("Invalid param 'index'")
 
-        return list(
-            self._validate_search_targets(
-                target for index in searchable_indexes for target in index.split(",")
-            )
-        )
+        generator = (target for index in searchable_indexes for target in index.split(","))
+        return list(self._validate_search_targets(generator, body))
 
     @staticmethod
     def _normalize_doc_type_to_list(doc_type):
diff --git a/tests/providers/elasticsearch/log/elasticmock/utilities/__init__.py b/tests/providers/elasticsearch/log/elasticmock/utilities/__init__.py
--- a/tests/providers/elasticsearch/log/elasticmock/utilities/__init__.py
+++ b/tests/providers/elasticsearch/log/elasticmock/utilities/__init__.py
@@ -39,13 +39,193 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 """Utilities for Elastic mock"""
+import base64
 import random
 import string
+from datetime import date, datetime
+from functools import wraps
+
+from elasticsearch.exceptions import NotFoundError
 
 DEFAULT_ELASTICSEARCH_ID_SIZE = 20
 CHARSET_FOR_ELASTICSEARCH_ID = string.ascii_letters + string.digits
+GLOBAL_PARAMS = ("pretty", "human", "error_trace", "format", "filter_path")
 
 
 def get_random_id(size=DEFAULT_ELASTICSEARCH_ID_SIZE):
     """Returns random if for elasticsearch"""
     return "".join(random.choice(CHARSET_FOR_ELASTICSEARCH_ID) for _ in range(size))
+
+
+def query_params(*es_query_params, **kwargs):
+    """
+    Decorator that pops all accepted parameters from method's kwargs and puts
+    them in the params argument.
+    """
+    body_params = kwargs.pop("body_params", None)
+    body_only_params = set(body_params or ()) - set(es_query_params)
+    body_name = kwargs.pop("body_name", None)
+    body_required = kwargs.pop("body_required", False)
+    type_possible_in_params = "type" in es_query_params
+
+    assert not (body_name and body_params)
+
+    assert not (body_name and body_required)
+    assert not body_required or body_params
+
+    def _wrapper(func):
+        @wraps(func)
+        def _wrapped(*args, **kwargs):
+            params = (kwargs.pop("params", None) or {}).copy()
+            headers = {k.lower(): v for k, v in (kwargs.pop("headers", None) or {}).copy().items()}
+
+            if "opaque_id" in kwargs:
+                headers["x-opaque-id"] = kwargs.pop("opaque_id")
+
+            http_auth = kwargs.pop("http_auth", None)
+            api_key = kwargs.pop("api_key", None)
+
+            using_body_kwarg = kwargs.get("body", None) is not None
+            using_positional_args = args and len(args) > 1
+
+            if type_possible_in_params:
+                doc_type_in_params = params and "doc_type" in params
+                doc_type_in_kwargs = "doc_type" in kwargs
+
+                if doc_type_in_params:
+                    params["type"] = params.pop("doc_type")
+                if doc_type_in_kwargs:
+                    kwargs["type"] = kwargs.pop("doc_type")
+
+            if using_body_kwarg or using_positional_args:
+                body_only_params_in_use = body_only_params.intersection(kwargs)
+                if body_only_params_in_use:
+                    params_prose = "', '".join(sorted(body_only_params_in_use))
+                    plural_params = len(body_only_params_in_use) > 1
+
+                    raise TypeError(
+                        f"The '{params_prose}' parameter{'s' if plural_params else ''} "
+                        f"{'are' if plural_params else 'is'} only serialized in the "
+                        f"request body and can't be combined with the 'body' parameter. "
+                        f"Either stop using the 'body' parameter and use keyword-arguments "
+                        f"only or move the specified parameters into the 'body'. "
+                        f"See https://github.com/elastic/elasticsearch-py/issues/1698 "
+                        f"for more information"
+                    )
+
+            elif set(body_params or ()).intersection(kwargs):
+                body = {}
+                for param in body_params:
+                    value = kwargs.pop(param, None)
+                    if value is not None:
+                        body[param.rstrip("_")] = value
+                kwargs["body"] = body
+
+            elif body_required:
+                kwargs["body"] = {}
+
+            if body_name:
+                if body_name in kwargs:
+                    if using_body_kwarg:
+                        raise TypeError(
+                            f"Can't use '{body_name}' and 'body' parameters together"
+                            f" because '{body_name}' is an alias for 'body'. "
+                            f"Instead you should only use the '{body_name}' "
+                            f"parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 "
+                            f"for more information"
+                        )
+                    kwargs["body"] = kwargs.pop(body_name)
+
+            if http_auth is not None and api_key is not None:
+                raise ValueError("Only one of 'http_auth' and 'api_key' may be passed at a time")
+            elif http_auth is not None:
+                headers["authorization"] = f"Basic {_base64_auth_header(http_auth)}"
+            elif api_key is not None:
+                headers["authorization"] = f"ApiKey {_base64_auth_header(api_key)}"
+
+            for p in es_query_params + GLOBAL_PARAMS:
+                if p in kwargs:
+                    v = kwargs.pop(p)
+                    if v is not None:
+                        params[p] = _escape(v)
+
+            for p in ("ignore", "request_timeout"):
+                if p in kwargs:
+                    params[p] = kwargs.pop(p)
+            return func(*args, params=params, headers=headers, **kwargs)
+
+        return _wrapped
+
+    return _wrapper
+
+
+def to_str(x, encoding="ascii"):
+    if not isinstance(x, str):
+        return x.decode(encoding)
+    return x
+
+
+def to_bytes(x, encoding="ascii"):
+    if not isinstance(x, bytes):
+        return x.encode(encoding)
+    return x
+
+
+def _base64_auth_header(auth_value):
+    """Takes either a 2-tuple or a base64-encoded string
+    and returns a base64-encoded string to be used
+    as an HTTP authorization header.
+    """
+    if isinstance(auth_value, (list, tuple)):
+        auth_value = base64.b64encode(to_bytes(":".join(auth_value)))
+    return to_str(auth_value)
+
+
+def _escape(value):
+    """
+    Escape a single value of a URL string or a query parameter. If it is a list
+    or tuple, turn it into a comma-separated string first.
+    """
+
+    # make sequences into comma-separated strings
+    if isinstance(value, (list, tuple)):
+        value = ",".join(value)
+
+    # dates and datetimes into isoformat
+    elif isinstance(value, (date, datetime)):
+        value = value.isoformat()
+
+    # make bools into true/false strings
+    elif isinstance(value, bool):
+        value = str(value).lower()
+
+    # don't decode bytestrings
+    elif isinstance(value, bytes):
+        return value
+
+    # encode strings to utf-8
+    if isinstance(value, str):
+        return value.encode("utf-8")
+
+    return str(value)
+
+
+class MissingIndexException(NotFoundError):
+    """Exception representing a missing index."""
+
+    def __init__(self, msg, body):
+        self.msg = msg
+        self.body = body
+
+    def __str__(self):
+        return f"IndexMissingException[[{self.msg}] missing] with body {self.body}"
+
+
+class SearchFailedException(NotFoundError):
+    """Exception representing a search failure."""
+
+    def __init__(self, msg):
+        self.msg = msg
+
+    def __str__(self):
+        return f"SearchFailedException: {self.msg}"
diff --git a/tests/providers/elasticsearch/log/test_es_task_handler.py b/tests/providers/elasticsearch/log/test_es_task_handler.py
--- a/tests/providers/elasticsearch/log/test_es_task_handler.py
+++ b/tests/providers/elasticsearch/log/test_es_task_handler.py
@@ -29,7 +29,6 @@
 import elasticsearch
 import pendulum
 import pytest
-from elasticsearch.exceptions import ElasticsearchException
 
 from airflow.configuration import conf
 from airflow.providers.elasticsearch.log.es_response import ElasticSearchResponse
@@ -40,6 +39,7 @@
 from tests.test_utils.db import clear_db_dags, clear_db_runs
 
 from .elasticmock import elasticmock
+from .elasticmock.utilities import SearchFailedException
 
 
 def get_ti(dag_id, task_id, execution_date, create_task_instance):
@@ -94,7 +94,7 @@ def setup_method(self, method):
             offset_field=self.offset_field,
         )
 
-        self.es = elasticsearch.Elasticsearch(hosts=[{"host": "localhost", "port": 9200}])
+        self.es = elasticsearch.Elasticsearch("http://localhost:9200")
         self.index_name = "test_index"
         self.doc_type = "log"
         self.test_message = "some random stuff"
@@ -132,7 +132,7 @@ def test_client(self):
     def test_client_with_config(self):
         es_conf = dict(conf.getsection("elasticsearch_configs"))
         expected_dict = {
-            "use_ssl": False,
+            "http_compress": False,
             "verify_certs": True,
         }
         assert es_conf == expected_dict
@@ -210,7 +210,7 @@ def test_read_with_patterns_no_match(self, ti):
     def test_read_with_missing_index(self, ti):
         ts = pendulum.now()
         with mock.patch.object(self.es_task_handler, "index_patterns", new="nonexistent,test_*"):
-            with pytest.raises(elasticsearch.exceptions.NotFoundError, match=r".*nonexistent.*"):
+            with pytest.raises(elasticsearch.exceptions.NotFoundError, match=r"IndexMissingException.*"):
                 self.es_task_handler.read(
                     ti, 1, {"offset": 0, "last_log_timestamp": str(ts), "end_of_log": False}
                 )
@@ -365,7 +365,7 @@ def test_read_as_download_logs(self, ti):
     def test_read_raises(self, ti):
         with mock.patch.object(self.es_task_handler.log, "exception") as mock_exception:
             with mock.patch.object(self.es_task_handler.client, "search") as mock_execute:
-                mock_execute.side_effect = ElasticsearchException("Failed to read")
+                mock_execute.side_effect = SearchFailedException("Failed to read")
                 logs, metadatas = self.es_task_handler.read(ti, 1)
             assert mock_exception.call_count == 1
             args, kwargs = mock_exception.call_args