Repository: elastic/elasticsearch-py

Issue:
elasticsearch-py 8.12 breaks the use of `from_` in body parameters
This bug was initially reported in the [Elastic Community Slack](https://www.elastic.co/blog/join-our-elastic-stack-workspace-on-slack). But first, come context.

## Context

Since the early days of the Elasticsearch Python client, [back in July 2013](https://github.com/elastic/elasticsearch-py/commit/48ec1ab4bbc0b49ac5dfcdd39fb341d93c7f4538), the `body` parameter is the way to specify the request body for requests that accept it. API calls using body look like this:

```python
es.search(index="bonsais", body={"query": {"match_all": {}}, "size": 50})
```

However, this parameter is an untyped Python dictionary which is not validated by the client. That said, thanks to the [Elasticsearch specification](https://github.com/elastic/elasticsearch-specification/) which provides the full types of each Elasticsearch API, we can provide a better experience. elasticsearch-py 8.0 did just that, introducing this new way of calling APIs, where the first level of body keys can be specified using Python parameters:

```python
es.search(index="bonsais", query={"match_all": {}}, size=50)
```

This has various advantages, including better autocompletion and type checks. For example, mypy will raise an error if size is not an integer. And since we realized we could [unpack](https://docs.python.org/3/tutorial/controlflow.html#unpacking-argument-lists) body to typed parameters like this:

```python
es.search(index="bonsais", **{"query": {"match_all": {}}, "size": 50})
```

We decided to deprecate the body API altogether. However, deprecating body has the following downsides:

* A lot of code written in the past decade was now triggering a deprecation warning
* Unknown parameters such as `sub_searches` or unintentional omissions from the Elasticsearch specification were rejected, causing queries to outright fail, unnecessarily forcing the use of raw requests.
* Optimizations such as passing an already encoded body to avoid paying the cost of serializing JSON were no longer possible.

The original author of the client, Honza Král, [pointed out those issues](https://github.com/elastic/elasticsearch-py/issues/2181), and we decided to allow `body` to work as before, without any warnings, [alongside the new API](https://github.com/elastic/elasticsearch-py/pull/2383). This is available elasticsearch-py 8.12.0.

## The case of Python keywords, like `from`

One subtlety with the above is that some identifiers are [reserved by Python](https://docs.python.org/3/reference/lexical_analysis.html#keywords) and can't be used as parameters. This is the case of `from`, for example. As such, `es.search(index="bonsais", query={"match_all": {}}, from=100, size=50)`, is invalid Python code. For this reason, parameter aliases were introduced, and the correct way to write that query was to use `from_`, eg. `es.search(index="bonsais", query={"match_all": {}}, from_=100, size=50)`. And then, under the hood, `from` is actually sent to Elasticsearch:

https://github.com/elastic/elasticsearch-py/blob/5014ce5337594f66040c81a2610220b1e8c0527e/elasticsearch/_sync/client/__init__.py#L1280-L1281

However, when the `body` parameter was deprecated in elasticsearch-py 8.12, it was deprecated by converting all `body` subfields to Python parameters internally, and *then* updated parameter aliases like `from_` to `from`. This means it was possible to write:

```python
es.search(index="bonsais", body={"query": {"match_all": {}}, "from_": 100, "size": 50})
```

which was then converted as if we had called:

```python
es.search(index="bonsais", query={"match_all": {}, from_=100, size=50)
```

to finally send `{"query": {"match_all": {}}, "from": 100, "size": 50}` as the body to Elasticsearch. This no longer works with elasticsearch-py 8.12.0. The body is used as is, without any inspection, and the correct way to use `from` with the `body` parameter is the one that always worked:

```python
es.search(
    index="*",
    body={
        "query": {"match_all": {}},
        "from": 10,
        "size": 10,
    },
)
```

I'm still not sure what the solution is here.

Comments:

Patch:
diff --git a/elasticsearch/_sync/client/utils.py b/elasticsearch/_sync/client/utils.py
index 4febb035..1d1a983a 100644
--- a/elasticsearch/_sync/client/utils.py
+++ b/elasticsearch/_sync/client/utils.py
@@ -298,7 +298,8 @@ def _merge_kwargs_no_duplicates(kwargs: Dict[str, Any], values: Dict[str, Any])
 
 def _merge_body_fields_no_duplicates(
     body: _TYPE_BODY, kwargs: Dict[str, Any], body_fields: Tuple[str, ...]
-) -> None:
+) -> bool:
+    mixed_body_and_params = False
     for key in list(kwargs.keys()):
         if key in body_fields:
             if isinstance(body, (str, bytes)):
@@ -315,11 +316,13 @@ def _merge_body_fields_no_duplicates(
             warnings.warn(
                 f"Received '{key}' via a specific parameter in the presence of a "
                 "'body' parameter, which is deprecated and will be removed in a future "
-                "version. Instead, use only 'body' or only specific paremeters.",
+                "version. Instead, use only 'body' or only specific parameters.",
                 category=DeprecationWarning,
                 stacklevel=warn_stacklevel(),
             )
             body[key] = kwargs.pop(key)
+            mixed_body_and_params = True
+    return mixed_body_and_params
 
 
 def _rewrite_parameters(
@@ -401,6 +404,7 @@ def _rewrite_parameters(
                 not ignore_deprecated_options or "body" not in ignore_deprecated_options
             ):
                 body: Optional[_TYPE_BODY] = kwargs.pop("body")
+                mixed_body_and_params = False
                 if body is not None:
                     if body_name:
                         if body_name in kwargs:
@@ -411,11 +415,27 @@ def _rewrite_parameters(
                                 "issues/1698 for more information"
                             )
                         kwargs[body_name] = body
-
                     elif body_fields is not None:
-                        _merge_body_fields_no_duplicates(body, kwargs, body_fields)
+                        mixed_body_and_params = _merge_body_fields_no_duplicates(
+                            body, kwargs, body_fields
+                        )
                         kwargs["body"] = body
 
+                    if parameter_aliases and not isinstance(body, (str, bytes)):
+                        for alias, rename_to in parameter_aliases.items():
+                            if rename_to in body:
+                                body[alias] = body.pop(rename_to)
+                                # If body and params are mixed, the alias may come from a param,
+                                # in which case the warning below will not make sense.
+                                if not mixed_body_and_params:
+                                    warnings.warn(
+                                        f"Using '{rename_to}' alias in 'body' is deprecated and will be removed "
+                                        f"in a future version of elasticsearch-py. Use '{alias}' directly instead. "
+                                        "See https://github.com/elastic/elasticsearch-py/issues/1698 for more information",
+                                        category=DeprecationWarning,
+                                        stacklevel=2,
+                                    )
+
             if parameter_aliases:
                 for alias, rename_to in parameter_aliases.items():
                     try:


Test patch:
diff --git a/test_elasticsearch/test_client/test_rewrite_parameters.py b/test_elasticsearch/test_client/test_rewrite_parameters.py
index 26218040..50a23256 100644
--- a/test_elasticsearch/test_client/test_rewrite_parameters.py
+++ b/test_elasticsearch/test_client/test_rewrite_parameters.py
@@ -191,7 +191,7 @@ class TestRewriteParameters:
         assert str(w[0].message) == (
             "Received 'source' via a specific parameter in the presence of a "
             "'body' parameter, which is deprecated and will be removed in a future "
-            "version. Instead, use only 'body' or only specific paremeters."
+            "version. Instead, use only 'body' or only specific parameters."
         )
 
     def test_body_fields_conflict(self):
@@ -238,6 +238,41 @@ class TestRewriteParameters:
         self.wrapped_func_aliases(source=["key3"])
         assert self.calls[-1] == ((), {"source": ["key3"]})
 
+    def test_parameter_aliases_body(self):
+        with pytest.warns(
+            DeprecationWarning,
+            match=(
+                "Using 'source' alias in 'body' is deprecated and will be removed in a future version of elasticsearch-py. "
+                "Use '_source' directly instead."
+            ),
+        ):
+            self.wrapped_func_aliases(body={"source": ["key4"]})
+
+        # using the correct name does not warn
+        with warnings.catch_warnings():
+            warnings.simplefilter("error")
+            self.wrapped_func_aliases(body={"_source": ["key4"]})
+
+    def test_parameter_aliases_body_param(self):
+        with pytest.warns(
+            DeprecationWarning,
+            match=(
+                "Received 'source' via a specific parameter in the presence of a "
+                "'body' parameter, which is deprecated and will be removed in a future "
+                "version. Instead, use only 'body' or only specific parameters."
+            ),
+        ):
+            self.wrapped_func_aliases(
+                source=["key4"], body={"query": {"match_all": {}}}
+            )
+
+        # using the correct name does not warn
+        with warnings.catch_warnings():
+            warnings.simplefilter("error")
+            self.wrapped_func_aliases(
+                body={"query": {"match_all": {}}, "_source": ["key4"]}
+            )
+
     @pytest.mark.parametrize("client_cls", [Elasticsearch, AsyncElasticsearch])
     def test_positional_argument_error(self, client_cls):
         client = client_cls("https://localhost:9200")