internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: elastic/apm-agent-python

Issue:
Performance Bug in the varmap function
**Describe the bug**: ...
When python code throws an error, and one of the local variables contains a very large list of lists of lists, it takes the elastic python apm client a very long time to process through the list. In our case it was taking over 60 seconds for the code to complete.


**To Reproduce**
To simply replicate the issue, I: (see code snipit  in the elastic forum, link below)

- created a simple script to create a list of list of lists of integers.
- in one method I had my "with elasticAPMTransaction:" statement.
- under this I created the list (200 integers) of list (300 integers) of list (40 integers) aka 24M items total.
- started to iterate through the outer list and picked an arbitrary spot to through an exception. This essentially mimicked the method where the real code threw an exception.
- wrapped this whole thing in a profiler and evaluated the output.
This would take 60 seconds +/- to complete.

Seems like the issue is in the varmap function in elasticapm\utils_init_.py but there maybe lots of ways to solve this issue.

**Environment (please complete the following information)**
- OS: [e.g. Linux]
- Python version: 3.x
- Framework and version [e.g. Django 2.1]:
- APM Server version: N/A
- Agent version: 6.7.2


**Additional context**
Created a case here https://discuss.elastic.co/t/python-client-apm-very-slow-when-local-list-variable-is-very-large/309935. See this for additional details.


Discussion:

Resolving patch:
diff --git a/elasticapm/utils/__init__.py b/elasticapm/utils/__init__.py
--- a/elasticapm/utils/__init__.py
+++ b/elasticapm/utils/__init__.py
@@ -64,10 +64,14 @@ def varmap(func, var, context=None, name=None, **kwargs):
         return func(name, "<...>", **kwargs)
     context.add(objid)
     if isinstance(var, dict):
+        # Apply func() before recursion, so that `shorten()` doesn't have to iterate over all the trimmed values
+        ret = func(name, var, **kwargs)
         # iterate over a copy of the dictionary to avoid "dictionary changed size during iteration" issues
-        ret = func(name, dict((k, varmap(func, v, context, k, **kwargs)) for k, v in var.copy().items()), **kwargs)
+        ret = dict((k, varmap(func, v, context, k, **kwargs)) for k, v in ret.copy().items())
     elif isinstance(var, (list, tuple)):
-        ret = func(name, [varmap(func, f, context, name, **kwargs) for f in var], **kwargs)
+        # Apply func() before recursion, so that `shorten()` doesn't have to iterate over all the trimmed values
+        ret = func(name, var, **kwargs)
+        ret = [varmap(func, f, context, name, **kwargs) for f in ret]
     else:
         ret = func(name, var, **kwargs)
     context.remove(objid)


Test patch:
Repository: elastic/apm-agent-python

Issue:
elasticapm.label fails when trying to warn that no transaction is present
**Describe the bug**:
https://github.com/elastic/apm-agent-python/blob/main/elasticapm/traces.py#L1092

When neither Transaction nor client is present, `elasticapm.label( . . .)` will fail while trying do print a warning.
This can happen for example when testing routes with fastapi.

Open Question would be whether the warning should be printed if no client is present.
I think it makes sense to do this since this is also the default from the config.

**To Reproduce**

```
import elasticapm

elasticapm.label(foo="foo")
```


**Environment (please complete the following information)**
- OS: Tested on Linux and Mac but should appear everywhere else
- Python version: Tested on 3.9/3.10
- Agent version: 6.10.1



Discussion:

Resolving patch:
diff --git a/elasticapm/traces.py b/elasticapm/traces.py
--- a/elasticapm/traces.py
+++ b/elasticapm/traces.py
@@ -1089,7 +1089,8 @@ def label(**labels):
     """
     transaction = execution_context.get_transaction()
     if not transaction:
-        if elasticapm.get_client().config.enabled:
+        client = elasticapm.get_client()
+        if not client or client.config.enabled:
             error_logger.warning("Ignored labels %s. No transaction currently active.", ", ".join(labels.keys()))
     else:
         transaction.label(**labels)


Test patch:
diff --git a/tests/client/client_tests.py b/tests/client/client_tests.py
--- a/tests/client/client_tests.py
+++ b/tests/client/client_tests.py
@@ -536,3 +536,7 @@ def test_check_server_version(elasticapm_client):
 )
 def test_user_agent(elasticapm_client, expected):
     assert elasticapm_client.get_user_agent() == "apm-agent-python/unknown (myapp{})".format(expected)
+
+
+def test_label_without_client():
+    elasticapm.label(foo="foo")
Repository: elastic/apm-agent-python

Issue:
FastAPI/Starlette: Truncate exception message to prevent "event exceeded the permitted size" error from APM Server
**Describe the bug**: 
The capture_exception() method is sending the whole exception message to the APM Server and when the message exceeds the maximum size, the APM Server returns the  "event exceeded the permitted size" error. 

I'm sorry if it is not a bug, but I believe that the apm-agent should truncate the message before sending it to the APM Server.


**To Reproduce**
Create a FastAPI route that always raise an error with a large exception message (as the example below) and it will show an "event exceeded the permitted size" error on the next attempt to send events to APM server

```
@app.get('/apm_test/')
async def apm_test():
    try:
        raise Exception('test' * 1000000)
    except Exception as e:
        apm_client.capture_exception()
    return PlainTextResponse('', 200)
```

**Environment**
- OS: ubuntu 20.04
- Python version: 3.9.13
- Framework and version: fastapi==0.79.0
- APM Server version: 8.3.0
- Agent version: 6.11.0




Discussion:
This is related to #305 -- we should definitely be truncating that field but haven't been able to come up with a great agent-wide solution so we've been playing whack-a-mole when these issues pop up. Thanks for the report!

Resolving patch:
diff --git a/elasticapm/events.py b/elasticapm/events.py
--- a/elasticapm/events.py
+++ b/elasticapm/events.py
@@ -33,7 +33,7 @@
 import sys
 
 from elasticapm.conf.constants import EXCEPTION_CHAIN_MAX_DEPTH
-from elasticapm.utils import varmap
+from elasticapm.utils import encoding, varmap
 from elasticapm.utils.encoding import keyword_field, shorten, to_unicode
 from elasticapm.utils.logging import get_logger
 from elasticapm.utils.stacks import get_culprit, get_stack_info, iter_traceback_frames
@@ -133,6 +133,8 @@ def capture(client, exc_info=None, **kwargs):
         else:
             message = "%s: %s" % (exc_type, to_unicode(exc_value)) if exc_value else str(exc_type)
 
+        message = encoding.long_field(message)
+
         data = {
             "id": "%032x" % random.getrandbits(128),
             "culprit": keyword_field(culprit),


Test patch:
diff --git a/tests/client/exception_tests.py b/tests/client/exception_tests.py
--- a/tests/client/exception_tests.py
+++ b/tests/client/exception_tests.py
@@ -397,3 +397,13 @@ def generate_uuid():
         generate_uuid()
     except Exception:
         elasticapm_client.capture_exception()
+
+
+def test_long_message(elasticapm_client):
+    try:
+        raise Exception("t" * 1000000)
+    except:
+        elasticapm_client.capture_exception()
+
+    exception = elasticapm_client.events[ERROR][0]
+    assert exception["exception"]["message"] == f'Exception: {"t"*9988}{"…"}'
Repository: elastic/apm-agent-python

Issue:
[META 679] Add support for ELB and Lambda URL metadata in lambda integrations
See meta issue for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/679



Discussion:

Resolving patch:
diff --git a/elasticapm/contrib/serverless/aws.py b/elasticapm/contrib/serverless/aws.py
--- a/elasticapm/contrib/serverless/aws.py
+++ b/elasticapm/contrib/serverless/aws.py
@@ -261,8 +261,7 @@ def set_metadata_and_context(self, coldstart: bool) -> None:
         elif self.source == "elb":
             elb_target_group_arn = self.event["requestContext"]["elb"]["targetGroupArn"]
             faas["trigger"]["type"] = "http"
-            faas["trigger"]["request_id"] = self.event["headers"]["x-amzn-trace-id"]
-            service_context["origin"] = {"name": elb_target_group_arn.split(":")[5]}
+            service_context["origin"] = {"name": elb_target_group_arn.split(":")[5].split("/")[1]}
             service_context["origin"]["id"] = elb_target_group_arn
             cloud_context["origin"] = {}
             cloud_context["origin"]["service"] = {"name": "elb"}


Test patch:
diff --git a/tests/contrib/serverless/aws_tests.py b/tests/contrib/serverless/aws_tests.py
--- a/tests/contrib/serverless/aws_tests.py
+++ b/tests/contrib/serverless/aws_tests.py
@@ -265,6 +265,7 @@ def test_func(event, context):
     assert transaction["context"]["request"]["method"] == "POST"
     assert transaction["context"]["request"]["headers"]
     assert transaction["context"]["response"]["status_code"] == 200
+    assert transaction["context"]["service"]["origin"]["name"] == "lambda-279XGJDqGZ5rsrHC2Fjr"
 
 
 def test_capture_serverless_s3(event_s3, context, elasticapm_client):
Repository: elastic/apm-agent-python

Issue:
[META 689] Ignore *principal* headers by default
See meta issue for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/689



Discussion:

Resolving patch:
diff --git a/elasticapm/conf/constants.py b/elasticapm/conf/constants.py
--- a/elasticapm/conf/constants.py
+++ b/elasticapm/conf/constants.py
@@ -31,16 +31,17 @@
 import decimal
 import re
 from collections import namedtuple
+from typing import Pattern
 
 
-def _starmatch_to_regex(pattern):
+def _starmatch_to_regex(pattern: str) -> Pattern:
     """
     This is a duplicate of starmatch_to_regex() in utils/__init__.py
 
     Duplication to avoid circular imports
     """
     options = re.DOTALL
-    # check if we are case sensitive
+    # check if we are case-sensitive
     if pattern.startswith("(?-i)"):
         pattern = pattern[5:]
     else:
@@ -98,6 +99,7 @@ def _starmatch_to_regex(pattern):
     "*credit*",
     "*card*",
     "*auth*",
+    "*principal*",
     "set-cookie",
 ]
 
@@ -107,12 +109,7 @@ def _starmatch_to_regex(pattern):
     SUCCESS="success", FAILURE="failure", UNKNOWN="unknown"
 )
 
-try:
-    # Python 2
-    LABEL_TYPES = (bool, int, long, float, decimal.Decimal)
-except NameError:
-    # Python 3
-    LABEL_TYPES = (bool, int, float, decimal.Decimal)
+LABEL_TYPES = (bool, int, float, decimal.Decimal)
 
 TRACESTATE = namedtuple("TRACESTATE", ["SAMPLE_RATE"])(SAMPLE_RATE="s")
 TRACE_CONTINUATION_STRATEGY = namedtuple("TRACE_CONTINUATION_STRATEGY", ["CONTINUE", "RESTART", "RESTART_EXTERNAL"])(
diff --git a/elasticapm/utils/__init__.py b/elasticapm/utils/__init__.py
--- a/elasticapm/utils/__init__.py
+++ b/elasticapm/utils/__init__.py
@@ -183,7 +183,7 @@ def read_pem_file(file_obj) -> bytes:
 
 def starmatch_to_regex(pattern: str) -> Pattern:
     options = re.DOTALL
-    # check if we are case sensitive
+    # check if we are case-sensitive
     if pattern.startswith("(?-i)"):
         pattern = pattern[5:]
     else:


Test patch:
diff --git a/tests/processors/tests.py b/tests/processors/tests.py
--- a/tests/processors/tests.py
+++ b/tests/processors/tests.py
@@ -63,6 +63,7 @@ def http_test_data():
                     "authorization": "bearer xyz",
                     "some-header": "some-secret-value",
                     "cookie": "foo=bar; baz=foo",
+                    "Ms-Client-Principal-Id": "foo",
                 },
                 "cookies": {
                     "foo": "bar",
@@ -85,6 +86,7 @@ def http_test_data():
                     "authorization": "bearer xyz",
                     "some-header": "some-secret-value",
                     "cookie": "foo=bar; baz=foo",
+                    "Ms-Client-Principal-Id": "foo",
                 },
             },
         }
@@ -296,6 +298,7 @@ def test_sanitize_http_headers(elasticapm_client, custom_header, http_test_data)
         "password": processors.MASK,
         "secret": processors.MASK,
         "authorization": processors.MASK,
+        "Ms-Client-Principal-Id": processors.MASK,
     }
     expected.update(custom_header)
     assert result["context"]["request"]["headers"] == expected
Repository: elastic/apm-agent-python

Issue:
KeyError: 'db' for DroppedSpan when running scan query on Elasticsearch
**Describe the bug**: ...

Elastic APM fails with a `KeyError: db key not found`.

The application where this happens is a Django project that stores/reads data from Elasticsearch. I have APM enable (APM server and Elasticsearch cluster are both running on Elastic Cloud). The library fails with the aforementioned error (shown in the snippet and screenshots below) while running a scan query on Elasticsearch. It looks like it's dropping some spans, which ends up in this case:

```
            hits = self._get_hits(result_data)
            if hits:
                span.context["db"]["rows_affected"] = hits
 
            return result_data
```

here's a screenshot of what I see on the APM Error page:

![image](https://user-images.githubusercontent.com/13816/200343191-fcff09cb-17f0-4f95-90db-f49fd1ab632d.png)

Few variables from the context:

![image](https://user-images.githubusercontent.com/13816/200343396-b95e0f90-a62e-4a4d-ad5d-87554b2a1d53.png)


**To Reproduce**

Unfortunately, I don't have a reproducible snippet. 

**Environment (please complete the following information)**
- OS: Linux (containerized)
- Python version: 3.9.15
- Framework and version [e.g. Django 2.1]:
- APM Server version: 7.17.4
- Agent version: 

```
$ pip freeze | grep elastic
django-elasticsearch-dsl==7.2.2
elastic-apm==6.13.1
elasticsearch==7.17.4
elasticsearch-dsl==7.4.0
```


**Additional context**

Add any other context about the problem here.




Discussion:

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/elasticsearch.py b/elasticapm/instrumentation/packages/elasticsearch.py
--- a/elasticapm/instrumentation/packages/elasticsearch.py
+++ b/elasticapm/instrumentation/packages/elasticsearch.py
@@ -163,7 +163,7 @@ def call(self, module, method, wrapped, instance, args, kwargs):
 
             hits = self._get_hits(result_data)
             if hits:
-                span.context["db"]["rows_affected"] = hits
+                span.update_context("db", {"rows_affected": hits})
 
             return result_data
 


Test patch:
Repository: elastic/apm-agent-python

Issue:
[META 724] Collect information on the agent installation method
See meta issue and spec for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/724
- Spec issue: https://github.com/elastic/apm/issues/725



Discussion:

Resolving patch:
diff --git a/elasticapm/base.py b/elasticapm/base.py
--- a/elasticapm/base.py
+++ b/elasticapm/base.py
@@ -109,6 +109,7 @@ def __init__(self, config=None, **inline):
         self._service_info = None
         # setting server_version here is mainly used for testing
         self.server_version = inline.pop("server_version", None)
+        self.activation_method = None
 
         self.check_python_version()
 
@@ -218,8 +219,6 @@ def __init__(self, config=None, **inline):
         if config.enabled:
             self.start_threads()
 
-        self.activation_method = None
-
         # Save this Client object as the global CLIENT_SINGLETON
         set_client(self)
 
@@ -346,13 +345,15 @@ def get_service_info(self):
             "name": keyword_field(self.config.service_name),
             "environment": keyword_field(self.config.environment),
             "version": keyword_field(self.config.service_version),
-            "agent": {"name": "python", "version": elasticapm.VERSION},
+            "agent": {"name": "python", "version": elasticapm.VERSION, "activation_method": "unknown"},
             "language": {"name": "python", "version": keyword_field(platform.python_version())},
             "runtime": {
                 "name": keyword_field(platform.python_implementation()),
                 "version": keyword_field(runtime_version),
             },
         }
+        if self.activation_method:
+            result["agent"]["activation_method"] = self.activation_method
         if self.config.framework_name:
             result["framework"] = {
                 "name": keyword_field(self.config.framework_name),


Test patch:
diff --git a/tests/client/client_tests.py b/tests/client/client_tests.py
--- a/tests/client/client_tests.py
+++ b/tests/client/client_tests.py
@@ -59,6 +59,7 @@ def test_service_info(elasticapm_client):
     assert service_info["environment"] == elasticapm_client.config.environment == "production"
     assert service_info["language"] == {"name": "python", "version": platform.python_version()}
     assert service_info["agent"]["name"] == "python"
+    assert service_info["agent"]["activation_method"] == "unknown"
 
 
 @pytest.mark.parametrize(
Repository: elastic/apm-agent-python

Issue:
[META 735] Change default APM server_url to 'http://127.0.0.1:8200' to avoid IPv6 ambiguity
See meta issue for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/735



Discussion:

Resolving patch:
diff --git a/elasticapm/conf/__init__.py b/elasticapm/conf/__init__.py
--- a/elasticapm/conf/__init__.py
+++ b/elasticapm/conf/__init__.py
@@ -550,7 +550,7 @@ class Config(_ConfigBase):
     secret_token = _ConfigValue("SECRET_TOKEN")
     api_key = _ConfigValue("API_KEY")
     debug = _BoolConfigValue("DEBUG", default=False)
-    server_url = _ConfigValue("SERVER_URL", default="http://localhost:8200", required=True)
+    server_url = _ConfigValue("SERVER_URL", default="http://127.0.0.1:8200", required=True)
     server_cert = _ConfigValue("SERVER_CERT", validators=[FileIsReadableValidator()])
     verify_server_cert = _BoolConfigValue("VERIFY_SERVER_CERT", default=True)
     use_certifi = _BoolConfigValue("USE_CERTIFI", default=True)


Test patch:
diff --git a/tests/contrib/django/django_tests.py b/tests/contrib/django/django_tests.py
--- a/tests/contrib/django/django_tests.py
+++ b/tests/contrib/django/django_tests.py
@@ -1270,7 +1270,7 @@ def test_settings_server_url_default():
     with override_settings(ELASTIC_APM={}):
         call_command("elasticapm", "check", stdout=stdout)
     output = stdout.getvalue()
-    assert "SERVER_URL http://localhost:8200 looks fine" in output
+    assert "SERVER_URL http://127.0.0.1:8200 looks fine" in output
 
 
 def test_settings_server_url_is_empty_string():
Repository: elastic/apm-agent-python

Issue:
KeyError:'db' when capturing elasticsearch span context
**Describe the bug**: ...

**To Reproduce**

1. Instrument via capture serverless
2. Perform an async_scan:
```python
async with AsyncElasticsearch(**elastic_context) as client:
    async for doc in async_scan(
          client=client,
          index='XXX',
          query=query,
          _source=['field']
      ):
```
![image](https://user-images.githubusercontent.com/37186808/219222266-597464b9-9c70-4cea-b5de-f2c4b570cc5f.png)


**Environment (please complete the following information)**
- OS: [e.g. Linux] debian:bullseye-slim (python:3.9-slim container image) x86
- Python version: 3.9
- Framework and version [e.g. Django 2.1]: N/A AWS Lambda Function
- APM Server version: 8.6.1
- Agent version: 6.14.0


**Additional context**

Add any other context about the problem here.

- Agent config options <!-- be careful not to post sensitive information -->
  <details>
    <summary>Click to expand</summary>

    ```
    ELASTIC_APM_LOG_LEVEL: warning
    ELASTIC_APM_LAMBDA_APM_SERVER: https://<elastic-cloud-hostname>:443
    ELASTIC_APM_SECRET_TOKEN: ${env:ELASTIC_APM_SECRET_TOKEN}
    ELASTIC_APM_ENVIRONMENT: ${self:provider.stage}
    ELASTIC_APM_SPAN_COMPRESSION_SAME_KIND_MAX_DURATION: 50ms
    ```
  </details>
- `requirements.txt`:
  <details>
    <summary>Click to expand</summary>

    ```toml
    [tool.poetry.dependencies]
    python = "^3.9"
    elasticsearch = "^8"
    elastic-apm = "^6.14"
    tomlkit = "^0.11.1"
    uvloop = "^0.17.0"
    PyYAML = "^6.0"
    aiobotocore = "^2.3"
    urllib3 = "^1.26.10"
    python-dateutil = "^2.8.2"
    pyjarm = "^0.0.5"
    aiohttp = "^3.8"
    multidict = "^6"
    ```
  </details>



Discussion:

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/asyncio/elasticsearch.py b/elasticapm/instrumentation/packages/asyncio/elasticsearch.py
--- a/elasticapm/instrumentation/packages/asyncio/elasticsearch.py
+++ b/elasticapm/instrumentation/packages/asyncio/elasticsearch.py
@@ -106,6 +106,6 @@ async def call(self, module, method, wrapped, instance, args, kwargs):
 
             hits = self._get_hits(result_data)
             if hits:
-                span.context["db"]["rows_affected"] = hits
+                span.update_context("db", {"rows_affected": hits})
 
             return result_data


Test patch:
Repository: elastic/apm-agent-python

Issue:
botocore instrumentation creating empty services
When using `boto3` to call out to services outside of the specially handled services (S3, SQS, DynamoDB, SNS), the `botocore` instrumentation creates spans with an incorrect `service.name`. This results in extra, empty services in both the service list and the service map.


Discussion:

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/botocore.py b/elasticapm/instrumentation/packages/botocore.py
--- a/elasticapm/instrumentation/packages/botocore.py
+++ b/elasticapm/instrumentation/packages/botocore.py
@@ -238,15 +238,15 @@ def modify_span_sqs_post(span: SpanType, args, kwargs, result):
                 span.add_link(tp)
 
 
-def handle_default(operation_name, service, instance, args, kwargs, destination):
+def handle_default(operation_name, service, instance, args, kwargs, context):
     span_type = "aws"
     span_subtype = service.lower()
     span_action = operation_name
 
-    destination["service"] = {"name": span_subtype, "resource": span_subtype, "type": span_type}
+    context["destination"]["service"] = {"name": span_subtype, "resource": span_subtype, "type": span_type}
 
     signature = f"{service}:{operation_name}"
-    return HandlerInfo(signature, span_type, span_subtype, span_action, destination)
+    return HandlerInfo(signature, span_type, span_subtype, span_action, context)
 
 
 handlers = {


Test patch: