Repository: elastic/apm-agent-python

Issue:
Add RequestId to botocore spans
**Is your feature request related to a problem? Please describe.**

When opening an issue with AWS Support, they will ask for the Request ID. If you don't have this ID then they won't help you. You are out of luck if you can't reproduce the issue. 

**Describe the solution you'd like**
I would like to add the RequestID somewhere on the span for the boto3 calls. I couldn't find somewhere obvious to put it in the Elastic Common Schema (it doesn't matter to me where its stored on the span). I am mostly interested in getting this ID for errors but it seems simple enough to add it for errors and successful calls. Something like this (from your botocore.py):

```
from botocore.exceptions import ClientError

def call(self, module, method, wrapped, instance, args, kwargs):
      ...
      with capture_span(
            handler_info.signature,
            span_type=handler_info.span_type,
            leaf=True,
            span_subtype=handler_info.span_subtype,
            span_action=handler_info.span_action,
            extra=handler_info.context,
        ) as span:
            if service in span_modifiers:
                span_modifiers[service](span, args, kwargs)
            try:
                response = wrapped(*args, **kwargs)
                add_response_meta_data(span, response)
                return response
            except ClientError as client_error:
                add_response_meta_data(span, client_error.response)
                raise
            


def add_response_meta_data(span, response):
    if response:
        request_id = response.get('ResponseMetadata', {}).get('RequestId')
        span.label(RequestId = request_id)
```

I can submit a PR and tests if that helps.

**Describe alternatives you've considered**
Its possible to get the request ID if you add very verbose logging (as described here: https://github.com/boto/boto3/issues/1143) but that creates a lot of noise in the logs.



Discussion:
Hmmm, it seems like a no-brainer that we should be collecting this information. For our lambda integration, we do collect the request ID and store it in `faas.execution` and `faas.trigger.request_id`. But I don't think we should be doing anything `faas`-related in non-lambda transactions/spans. I'm going to ask around if other agents think we should get this added to intake -- if not, we can use labels as you've proposed.
I opened https://github.com/elastic/apm-server/issues/7871 to discuss adding `span.context.http.request.id` to the intake, which I think would be a great location for this Request ID. We could also potentially stick it in `span.context.response.headers` which is already supported in the intake. Stay tuned.
Thanks I really appreciate it!

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/asyncio/aiobotocore.py b/elasticapm/instrumentation/packages/asyncio/aiobotocore.py
--- a/elasticapm/instrumentation/packages/asyncio/aiobotocore.py
+++ b/elasticapm/instrumentation/packages/asyncio/aiobotocore.py
@@ -53,4 +53,7 @@ async def call(self, module, method, wrapped, instance, args, kwargs):
             result = await wrapped(*args, **kwargs)
             if service in post_span_modifiers:
                 post_span_modifiers[service](span, args, kwargs, result)
+            request_id = result.get("ResponseMetadata", {}).get("RequestId")
+            if request_id:
+                span.update_context("http", {"request": {"id": request_id}})
             return result
diff --git a/elasticapm/instrumentation/packages/botocore.py b/elasticapm/instrumentation/packages/botocore.py
--- a/elasticapm/instrumentation/packages/botocore.py
+++ b/elasticapm/instrumentation/packages/botocore.py
@@ -106,6 +106,9 @@ def call(self, module, method, wrapped, instance, args, kwargs):
             result = wrapped(*args, **kwargs)
             if service in post_span_modifiers:
                 post_span_modifiers[service](span, args, kwargs, result)
+            request_id = result.get("ResponseMetadata", {}).get("RequestId")
+            if request_id:
+                span.update_context("http", {"request": {"id": request_id}})
             return result
 
 


Test patch:
diff --git a/tests/instrumentation/botocore_tests.py b/tests/instrumentation/botocore_tests.py
--- a/tests/instrumentation/botocore_tests.py
+++ b/tests/instrumentation/botocore_tests.py
@@ -97,6 +97,7 @@ def test_botocore_instrumentation(instrument, elasticapm_client):
     assert span["type"] == "aws"
     assert span["subtype"] == "ec2"
     assert span["action"] == "DescribeInstances"
+    assert span["context"]["http"]["request"]["id"]
 
 
 def test_s3(instrument, elasticapm_client):
@@ -117,6 +118,7 @@ def test_s3(instrument, elasticapm_client):
         assert span["context"]["destination"]["service"]["name"] == "s3"
         assert span["context"]["destination"]["service"]["resource"] == "xyz"
         assert span["context"]["destination"]["service"]["type"] == "storage"
+        assert span["context"]["http"]["request"]["id"]
     assert spans[0]["name"] == "S3 CreateBucket xyz"
     assert spans[0]["action"] == "CreateBucket"
     assert spans[1]["name"] == "S3 PutObject xyz"
@@ -175,6 +177,7 @@ def test_dynamodb(instrument, elasticapm_client, dynamodb):
         assert span["context"]["destination"]["service"]["name"] == "dynamodb"
         assert span["context"]["destination"]["service"]["resource"] == "Movies"
         assert span["context"]["destination"]["service"]["type"] == "db"
+        assert span["context"]["http"]["request"]["id"]
     assert spans[0]["name"] == "DynamoDB PutItem Movies"
     assert spans[1]["name"] == "DynamoDB Query Movies"
     assert spans[1]["context"]["db"]["statement"] == "title = :v1 and #y = :v2"
@@ -200,6 +203,7 @@ def test_sns(instrument, elasticapm_client):
     assert spans[2]["context"]["destination"]["service"]["name"] == "sns"
     assert spans[2]["context"]["destination"]["service"]["resource"] == "sns/mytopic"
     assert spans[2]["context"]["destination"]["service"]["type"] == "messaging"
+    assert spans[2]["context"]["http"]["request"]["id"]
 
 
 def test_sqs_send(instrument, elasticapm_client, sqs_client_and_queue):
@@ -222,6 +226,7 @@ def test_sqs_send(instrument, elasticapm_client, sqs_client_and_queue):
     assert span["context"]["destination"]["service"]["name"] == "sqs"
     assert span["context"]["destination"]["service"]["resource"] == "sqs/myqueue"
     assert span["context"]["destination"]["service"]["type"] == "messaging"
+    assert span["context"]["http"]["request"]["id"]
 
     messages = sqs.receive_message(
         QueueUrl=queue_url,