Repository: elastic/apm-agent-python

Issue:
Missing header normalization on AWS lambda
**Describe the bug**: 

It looks we are missing some case normalization in the AWS Lambda instrumentation. Even if headers are documented as being already lowercase in [AWS docs](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html), it looks like there are cases where this is not.

Nodejs agent already handles the issue here:
https://github.com/elastic/apm-agent-nodejs/blob/2b69514d0b2c97bb974346cce975cf0cc958d559/lib/lambda.js#L705-L725

Outcome of this is that we are not getting any traceparent or tracestate header and so we are not able to correlate transactions.

**To Reproduce**

1. run apm agent instrumentation for aws lambda
2. inspect the event and see headers are not normalized
3. see that we are not able to get traceparent or tracestate header


**Environment (please complete the following information)**
- OS: [e.g. Linux] aws lambda
- Python version: -
- Framework and version [e.g. Django 2.1]: -
- APM Server version: -
- Agent version: latest
Missing header normalization on AWS lambda
**Describe the bug**: 

It looks we are missing some case normalization in the AWS Lambda instrumentation. Even if headers are documented as being already lowercase in [AWS docs](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html), it looks like there are cases where this is not.

Nodejs agent already handles the issue here:
https://github.com/elastic/apm-agent-nodejs/blob/2b69514d0b2c97bb974346cce975cf0cc958d559/lib/lambda.js#L705-L725

Outcome of this is that we are not getting any traceparent or tracestate header and so we are not able to correlate transactions.

**To Reproduce**

1. run apm agent instrumentation for aws lambda
2. inspect the event and see headers are not normalized
3. see that we are not able to get traceparent or tracestate header


**Environment (please complete the following information)**
- OS: [e.g. Linux] aws lambda
- Python version: -
- Framework and version [e.g. Django 2.1]: -
- APM Server version: -
- Agent version: latest


Discussion:

Resolving patch:
diff --git a/elasticapm/contrib/serverless/aws.py b/elasticapm/contrib/serverless/aws.py
--- a/elasticapm/contrib/serverless/aws.py
+++ b/elasticapm/contrib/serverless/aws.py
@@ -135,6 +135,18 @@ def prep_kwargs(kwargs=None):
     return kwargs
 
 
+def should_normalize_headers(event: dict) -> bool:
+    """
+    Helper to decide if we should normalize headers or not depending on the event
+
+    Even if the documentation says that headers are lowercased it's not always the case for format version 1.0
+    https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html
+    """
+
+    request_context = event.get("requestContext", {})
+    return ("elb" in request_context or "requestId" in request_context) and "http" not in request_context
+
+
 class _lambda_transaction(object):
     """
     Context manager for creating transactions around AWS Lambda functions.
@@ -162,7 +174,13 @@ def __enter__(self):
             # service like Step Functions, and is unlikely to be standardized
             # in any way. We just have to rely on our defaults in this case.
             self.event = {}
-        trace_parent = TraceParent.from_headers(self.event.get("headers") or {})
+
+        headers = self.event.get("headers") or {}
+        if headers and should_normalize_headers(self.event):
+            normalized_headers = {k.lower(): v for k, v in headers.items()}
+        else:
+            normalized_headers = headers
+        trace_parent = TraceParent.from_headers(normalized_headers)
 
         global COLD_START
         cold_start = COLD_START


Test patch:
diff --git a/tests/contrib/serverless/aws_elb_test_data.json b/tests/contrib/serverless/aws_elb_test_data.json
--- a/tests/contrib/serverless/aws_elb_test_data.json
+++ b/tests/contrib/serverless/aws_elb_test_data.json
@@ -15,6 +15,7 @@
         "connection": "Keep-Alive",
         "host": "blabla.com",
         "user-agent": "Apache-HttpClient/4.5.13 (Java/11.0.15)",
+        "TraceParent": "00-12345678901234567890123456789012-1234567890123456-01",
         "x-amzn-trace-id": "Root=1-xxxxxxxxxxxxxx",
         "x-forwarded-for": "199.99.99.999",
         "x-forwarded-port": "443",
diff --git a/tests/contrib/serverless/aws_tests.py b/tests/contrib/serverless/aws_tests.py
--- a/tests/contrib/serverless/aws_tests.py
+++ b/tests/contrib/serverless/aws_tests.py
@@ -36,7 +36,12 @@
 
 from elasticapm import capture_span
 from elasticapm.conf import constants
-from elasticapm.contrib.serverless.aws import capture_serverless, get_data_from_request, get_data_from_response
+from elasticapm.contrib.serverless.aws import (
+    capture_serverless,
+    get_data_from_request,
+    get_data_from_response,
+    should_normalize_headers,
+)
 
 
 @pytest.fixture
@@ -300,6 +305,7 @@ def test_func(event, context):
     assert transaction["context"]["request"]["headers"]
     assert transaction["context"]["response"]["status_code"] == 200
     assert transaction["context"]["service"]["origin"]["name"] == "lambda-279XGJDqGZ5rsrHC2Fjr"
+    assert transaction["trace_id"] == "12345678901234567890123456789012"
 
 
 def test_capture_serverless_s3(event_s3, context, elasticapm_client):
@@ -477,3 +483,17 @@ def test_func(event, context):
 
     test_func(event_api2, context)
     assert len(elasticapm_client.events[constants.TRANSACTION]) == 1
+
+
+def test_should_normalize_headers_true(event_api, event_elb):
+    assert should_normalize_headers(event_api) is True
+    assert should_normalize_headers(event_elb) is True
+
+
+def test_should_normalize_headers_false(event_api2, event_lurl, event_s3, event_s3_batch, event_sqs, event_sns):
+    assert should_normalize_headers(event_api2) is False
+    assert should_normalize_headers(event_lurl) is False
+    assert should_normalize_headers(event_s3) is False
+    assert should_normalize_headers(event_s3_batch) is False
+    assert should_normalize_headers(event_sqs) is False
+    assert should_normalize_headers(event_sns) is False