Repository: elastic/apm-agent-python

Issue:
AttributeError: 'Retry' object has no attribute 'copy'
The `elasticapm` Python client fails with `AttributeError: 'Retry' object has no attribute 'copy'` during HTTP retries.

The Elastic APM UI points towards the following lines being responsible for the crash https://github.com/elastic/apm-agent-python/blob/c9a1f7a74b4b8e39af06f9b383037a3ae49ca42b/elasticapm/instrumentation/packages/urllib3.py#L66-L68 

**To Reproduce**

Configure a `requests` session with [urllib3 retries](https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html) as described below. Then point it at a flaky endpoint.

``` python
class TimeoutHTTPAdapter(HTTPAdapter):
    def __init__(self, retries: int, *args: Any, **kwargs: Any) -> None:
        self.timeout = kwargs.pop("timeout", 30)
        self._retries = retries
        super().__init__(*args, **kwargs)

    def send( 
        self, request: requests.PreparedRequest, **kwargs: Any
    ) -> requests.Response:
        kwargs.setdefault("timeout", self.timeout)
        return super().send(request, **kwargs)

def session_with_retries(
    retries: int = 3,
    backoff_factor: float = 1,
    timeout: int = 30,
    status_forcelist: Collection[int] = (408, 429, 500, 502, 503, 504),
) -> requests.Session:

    retry_strategy = Retry(
        total=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        allowed_methods=Retry.DEFAULT_ALLOWED_METHODS.union(["POST", "PATCH"]),
    )
    adapter = TimeoutHTTPAdapter(
        retries=retries, max_retries=retry_strategy, timeout=timeout
    )

    s = requests.Session()
    s.mount("http://", adapter)
    s.mount("https://", adapter)
    return s
```

**Environment**
- OS: Linux
- Python version: Python 3.11
- Framework and version: `urllib3==2.0.6` and `requests==2.31.0`
- APM Server version: 8.10.4
- Agent version: `elastic-apm==6.19.0`

**Background**
* This appears to be related to the recent urllib3 fixes https://github.com/elastic/apm-agent-python/pull/1822 and https://github.com/elastic/apm-agent-python/issues/1816 
AttributeError: 'Retry' object has no attribute 'copy'
The `elasticapm` Python client fails with `AttributeError: 'Retry' object has no attribute 'copy'` during HTTP retries.

The Elastic APM UI points towards the following lines being responsible for the crash https://github.com/elastic/apm-agent-python/blob/c9a1f7a74b4b8e39af06f9b383037a3ae49ca42b/elasticapm/instrumentation/packages/urllib3.py#L66-L68 

**To Reproduce**

Configure a `requests` session with [urllib3 retries](https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html) as described below. Then point it at a flaky endpoint.

``` python
class TimeoutHTTPAdapter(HTTPAdapter):
    def __init__(self, retries: int, *args: Any, **kwargs: Any) -> None:
        self.timeout = kwargs.pop("timeout", 30)
        self._retries = retries
        super().__init__(*args, **kwargs)

    def send( 
        self, request: requests.PreparedRequest, **kwargs: Any
    ) -> requests.Response:
        kwargs.setdefault("timeout", self.timeout)
        return super().send(request, **kwargs)

def session_with_retries(
    retries: int = 3,
    backoff_factor: float = 1,
    timeout: int = 30,
    status_forcelist: Collection[int] = (408, 429, 500, 502, 503, 504),
) -> requests.Session:

    retry_strategy = Retry(
        total=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
        allowed_methods=Retry.DEFAULT_ALLOWED_METHODS.union(["POST", "PATCH"]),
    )
    adapter = TimeoutHTTPAdapter(
        retries=retries, max_retries=retry_strategy, timeout=timeout
    )

    s = requests.Session()
    s.mount("http://", adapter)
    s.mount("https://", adapter)
    return s
```

**Environment**
- OS: Linux
- Python version: Python 3.11
- Framework and version: `urllib3==2.0.6` and `requests==2.31.0`
- APM Server version: 8.10.4
- Agent version: `elastic-apm==6.19.0`

**Background**
* This appears to be related to the recent urllib3 fixes https://github.com/elastic/apm-agent-python/pull/1822 and https://github.com/elastic/apm-agent-python/issues/1816 


Discussion:
hello all, can you please suggest older previous version of elastic apm agent which is stable and does not contain that issue?
@cluster-master  You need to use an older version of `urllib3`; no version of the agent has a fix for this yet.
@StephanErb could you please provide an example requests session method call please? Just need the method and what parameters you are passing, no need for actual values. Thanks.
You can find a public example here which triggers the issue: https://github.com/kubernetes-client/python/blob/master/kubernetes/client/rest.py#L217-L221 

I don't seem to be able to reproduce it with anything else any longer besides the Kubernetes client. 
@StephanErb thanks!
@StephanErb the kubernetes code pass args as kwargs though so it'll not pass thru the same code path. Do you have another stack trace? Or maybe you have an older kubernetes client version?

@xrmx does that help? 
```
kubernetes==27.2.0
urllib3==2.2.1
elastic-apm==6.20.0
```
![Elastic-APM-Error](https://github.com/elastic/apm-agent-python/assets/483730/8e7e4fd2-98b2-4f4b-8fcd-7a1286da9d7a)


@StephanErb that's useful, thanks! Looks like I really need a TimeoutError and not having a connection.
hello all, can you please suggest older previous version of elastic apm agent which is stable and does not contain that issue?
@cluster-master  You need to use an older version of `urllib3`; no version of the agent has a fix for this yet.
@StephanErb could you please provide an example requests session method call please? Just need the method and what parameters you are passing, no need for actual values. Thanks.
You can find a public example here which triggers the issue: https://github.com/kubernetes-client/python/blob/master/kubernetes/client/rest.py#L217-L221 

I don't seem to be able to reproduce it with anything else any longer besides the Kubernetes client. 
@StephanErb thanks!
@StephanErb the kubernetes code pass args as kwargs though so it'll not pass thru the same code path. Do you have another stack trace? Or maybe you have an older kubernetes client version?

@xrmx does that help? 
```
kubernetes==27.2.0
urllib3==2.2.1
elastic-apm==6.20.0
```
![Elastic-APM-Error](https://github.com/elastic/apm-agent-python/assets/483730/8e7e4fd2-98b2-4f4b-8fcd-7a1286da9d7a)


@StephanErb that's useful, thanks! Looks like I really need a TimeoutError and not having a connection.

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/urllib3.py b/elasticapm/instrumentation/packages/urllib3.py
--- a/elasticapm/instrumentation/packages/urllib3.py
+++ b/elasticapm/instrumentation/packages/urllib3.py
@@ -61,12 +61,7 @@ def update_headers(args, kwargs, instance, transaction, trace_parent):
     :param trace_parent: the TraceParent object
     :return: an (args, kwargs) tuple
     """
-    from urllib3._version import __version__ as urllib3_version
-
-    if urllib3_version.startswith("2") and len(args) >= 5 and args[4]:
-        headers = args[4].copy()
-        args = tuple(itertools.chain((args[:4]), (headers,), args[5:]))
-    elif len(args) >= 4 and args[3]:
+    if len(args) >= 4 and args[3]:
         headers = args[3].copy()
         args = tuple(itertools.chain((args[:3]), (headers,), args[4:]))
     elif "headers" in kwargs and kwargs["headers"]:


Test patch:
diff --git a/tests/instrumentation/urllib3_tests.py b/tests/instrumentation/urllib3_tests.py
--- a/tests/instrumentation/urllib3_tests.py
+++ b/tests/instrumentation/urllib3_tests.py
@@ -294,3 +294,20 @@ def test_instance_headers_are_respected(
         assert "kwargs" in request_headers
     if instance_headers and not (header_arg or header_kwarg):
         assert "instance" in request_headers
+
+
+def test_connection_pool_urlopen_does_not_crash_with_many_args(instrument, elasticapm_client, waiting_httpserver):
+    """Mimics ConnectionPool.urlopen error path with broken connection, see #1928"""
+    waiting_httpserver.serve_content("")
+    url = waiting_httpserver.url + "/hello_world"
+    parsed_url = urllib.parse.urlparse(url)
+    pool = urllib3.HTTPConnectionPool(
+        parsed_url.hostname,
+        parsed_url.port,
+        maxsize=1,
+        block=True,
+    )
+    retry = urllib3.util.Retry(10)
+    elasticapm_client.begin_transaction("transaction")
+    r = pool.urlopen("GET", url, None, {"args": "true"}, retry, False, False)
+    assert r.status == 200