Repository: elastic/apm-agent-python

Issue:
Elastic APM crashes when trying to inspect hits from queries while track_total_hits is False
**Describe the bug**: 

The Elastic APM instrumentation for _Elasticsearch_ tries to get total number of hits in the results of a query but if the **track_total_hits** is set to **False** the `result.body['hits']` does not have a **total** key.

Fails at this line exactly with a KeyError('total'): https://github.com/elastic/apm-agent-python/blob/8b22d6a910ceda6587b99e47d8b3c9fa95c8d490/elasticapm/instrumentation/packages/elasticsearch.py#L180

**To Reproduce**

1. Instrument your python code with ElasticAPM, you don't even need to have the APM server up it crashes while collecting the data
2. Run any search query with **track_total_hits** set to **False**
`client.search(params={'track_total_hits': False}, **query_body)`

**Environment (please complete the following information)**
- OS: Linux
- Python version: 3.9
- Framework and version [e.g. Django 2.1]: Flask
- APM Server version: Irrelevent
- Agent version: 6.13.0


**Additional context**

This will crash no matter the config. We've reverted our code to use ElasticAPM 6.12 to make it work again.


Discussion:

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/elasticsearch.py b/elasticapm/instrumentation/packages/elasticsearch.py
--- a/elasticapm/instrumentation/packages/elasticsearch.py
+++ b/elasticapm/instrumentation/packages/elasticsearch.py
@@ -177,8 +177,8 @@ def _get_signature(self, args, kwargs):
 
     def _get_hits(self, result) -> Optional[int]:
         if getattr(result, "body", None) and "hits" in result.body:  # ES >= 8
-            return result.body["hits"]["total"]["value"]
-        elif isinstance(result, dict) and "hits" in result:
+            return result.body["hits"].get("total", {}).get("value")
+        elif isinstance(result, dict) and "hits" in result and "total" in result["hits"]:
             return (
                 result["hits"]["total"]["value"]
                 if isinstance(result["hits"]["total"], dict)


Test patch:
diff --git a/tests/instrumentation/elasticsearch_tests.py b/tests/instrumentation/elasticsearch_tests.py
--- a/tests/instrumentation/elasticsearch_tests.py
+++ b/tests/instrumentation/elasticsearch_tests.py
@@ -399,6 +399,32 @@ def test_search_body(instrument, elasticapm_client, elasticsearch):
     assert span["context"]["http"]["status_code"] == 200
 
 
+@pytest.mark.skipif(ES_VERSION[0] < 7, reason="track_total_hits unsupported")
+@pytest.mark.integrationtest
+def test_search_track_total_hits_false(instrument, elasticapm_client, elasticsearch):
+    elasticsearch.create(
+        index="tweets", id="1", refresh=True, **get_kwargs({"user": "kimchy", "text": "hola", "userid": 1})
+    )
+    elasticapm_client.begin_transaction("test")
+    search_query = {"query": {"term": {"user": "kimchy"}}, "sort": ["userid"]}
+    result = elasticsearch.search(body=search_query, track_total_hits=False)
+    elasticapm_client.end_transaction("test", "OK")
+
+    transaction = elasticapm_client.events[TRANSACTION][0]
+    assert result["hits"]["hits"][0]["_source"] == {"user": "kimchy", "text": "hola", "userid": 1}
+    spans = elasticapm_client.spans_for_transaction(transaction)
+    assert len(spans) == 1
+    span = spans[0]
+    # Depending on ES_VERSION, could be /_all/_search or /_search, and GET or POST
+    assert span["name"] in ("ES GET /_search", "ES GET /_all/_search", "ES POST /_search")
+    assert span["type"] == "db"
+    assert span["subtype"] == "elasticsearch"
+    assert span["action"] == "query"
+    assert span["context"]["db"]["type"] == "elasticsearch"
+    assert "rows_affected" not in span["context"]["db"]
+    assert span["context"]["http"]["status_code"] == 200
+
+
 @pytest.mark.integrationtest
 def test_search_querystring(instrument, elasticapm_client, elasticsearch):
     elasticsearch.create(index="tweets", id="1", refresh=True, **get_kwargs({"user": "kimchy", "text": "hola"}))