Repository: elastic/apm-agent-python

Issue:
Frequent exception in interval timer function with SSL error
**Describe the bug**:

We are getting frequent weird SSL errors in our Celery tasks that we cannot track down.

```
Exception in interval timer function

Failed to submit message: "Unable to reach APM Server: [('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')] (url: https://XXXX.apm.eu-central-1.aws.cloud.es.io:443/intake/v2/events)"

Error: [('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')]
  File "elasticapm/utils/threading.py", line 84, in run
    rval = self._function(*self._args, **self._kwargs)
  File "elasticapm/conf/__init__.py", line 743, in update_config
    new_version, new_config, next_run = self.transport.get_config(self.config_version, keys)
  File "elasticapm/transport/http.py", line 141, in get_config
    response = self.http.urlopen(
  File "urllib3/poolmanager.py", line 324, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "elasticapm/instrumentation/packages/base.py", line 205, in call_if_sampling
    return wrapped(*args, **kwargs)
  File "urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File "urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
    
  File "urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "http/client.py", line 1371, in getresponse
    response.begin()
  File "http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "urllib3/contrib/pyopenssl.py", line 297, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "OpenSSL/SSL.py", line 1822, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "OpenSSL/SSL.py", line 1647, in _raise_ssl_error
    _raise_current_error()
  File "OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
```

The APM server is reachable and according to Sentry traces, calls just seconds before and after are working fine.
Unfortunately these errors correlate with Celery tasks having transactions problems (waiting for database locks forever).
We tried upgrading dependencies and the underlying Docker container to now Python 3.9 but that didn't help.

**Environment (please complete the following information)**
- OS: Docker image `python:3.9`
- Python version: 3.9.7 (default, Oct 13 2021, 09:00:49) [GCC 10.2.1 20210110])
- Framework and version: 3.2.8
- APM Server version: v7.13.2
- Agent version: 6.6.0


**Additional context**

Add any other context about the problem here.

- Agent config options <!-- be careful not to post sensitive information -->
  <details>
    <summary>Click to expand</summary>

    ```
    ELASTIC_APM = {
    "SERVER_URL": ELASTIC_APM_SERVER_URL,
    "SECRET_TOKEN": ssm_get("elastic-apm-secret-token", STACK_NAME),
    "SERVICE_NAME": env("STACK_NAME"),
    "ENVIRONMENT": env("STACK_NAME"),
    "SERVICE_VERSION": env("GIT_VERSION"),
    "TRANSACTION_SAMPLE_RATE": 0.2,
    "TRANSACTION_NAME_FROM_ROUTE": True,
    "TRANSACTION_MAX_SPANS": 50,
    "API_REQUEST_TIME": "5s",
    "CLOUD_PROVIDER": False,
    "TRANSACTIONS_IGNORE_PATTERNS": [
        "submissions.tasks.check_submissions_periodically_task",
        "base.views.HealthCheckCustomView",
        "base.views.okay_view",
        "health_check.contrib.celery.tasks.add",
    ],
    }
    NEW_APPS += ["elasticapm.contrib.django"]
    ```
  </details>
- `requirements.txt`:
  <details>
    <summary>Click to expand</summary>

    ```
        [[source]]
        url = "https://pypi.org/simple"
        verify_ssl = true
        name = "pypi"
        
        [packages]
        argon2_cffi = "~=20.1.0"
        attrs = "~=19.3.0"
        bagit = "~=1.7.0"
        canonicaljson = "~=1.2.0"
        celery = "~=5.1.2"
        celery-redbeat = "~=2.0.0"
        Django = "~=3.2.8"
        django-environ = "~=0.4.5"
        django-extensions = "~=3.0.9"
        django-model-utils = "~=4.0.0"
        django-redis = "~=5.0.0"
        django-storages = "~=1.11.1"
        djangorestframework = "~=3.12.4"
        elastic-apm = "~=6.6.0"
        elasticsearch = "<7.0.0"
        elasticsearch-dsl = "<7.0.0"
        gevent = "~=21.1.2"
        kombu = "~=5.1.0"
        mock = "~=4.0.2"
        model_mommy = "~=2.0.0"
        Pillow = "~=8.2.0"
        psycopg2 = "~=2.9.1"
        pycountry = "~=20.7"
        PyYAML = "~=5.4"
        redis = "~=3.5.3"
        requests = "~=2.26"
        sentry-sdk = "~=1.4.3"
        ssm-cache = "~=2.5"
        uWSGI = "~=2.0.19"
        word2number = "~=1.1"
        
        [dev-packages]
        autoflake = "~=1.4"
        black = "==21.9b0"
        coverage = "~=5.2"
        ddt = "~=1.4"
        django-cprofile-middleware = "*"
        django-debug-toolbar = "~=2.2"
        django-jenkins = "~=0.110"
        django-webtest = "~=1.9"
        docformatter = "~=1.0"
        httmock = "~=1.2"
        hypothesis = "~=5.23"
        ipdb = "~=0.13"
        isort = "~=5.5"
        mypy = "~=0.782"
        pep8 = "~=1.7"
        pep8-naming = "~=0.11"
        pydocstyle = "~=5.0"
        tblib = "~=1.7"
        yapf = "~=0.30"
        
        [requires]
        python_version = "3.9"

    ```
  </details>



Discussion:
Hey @webjunkie. The error "decryption failed or bad record mac" seems to be often related to an SSL connection being in a bad state after a process fork. We had a similar report in the past, but never managed to track down the exact problem.

OpenSSL itself should be fork-safe since version `1.1.0c`. I'm pretty sure you have a newer version bundled with Python 3.9, nevertheless, could you give us the output from this code in your environment where you see the error?

    import ssl
    print(ssl.OPENSSL_VERSION)

Do you only see this problem in Celery, or also in your webapp? And do you use celery with pre-fork workers or something else? (gevent, eventlet etc.)?
Thanks for the quick reply! 

The output is `OpenSSL 1.1.1k  25 Mar 2021`.

We use Celery with prefork/default workers. We used to have gevent, but switched due to other issues I can't exactly remember anymore. Coincidentally, I can see in Sentry the SSL errors appearing first when we did the switch to prefork. This was a while ago and we treated the SSL errors as noise until we now have more problems with our tasks and we found they are possibly related. We never saw these errors in regular webapp requests.
Let me know if you need any other details.
The fact that this only showed up once you started using pre-fork workers is super interesting! A similar issue seems to happen with uwsgi in pre-fork mode (see e.g. [this stackoverflow thread](https://stackoverflow.com/questions/22752521/uwsgi-flask-sqlalchemy-and-postgres-ssl-error-decryption-failed-or-bad-reco)).

My theory is that we open an HTTP connection before celery gets around to fork the worker processes, which then leads to both the main process and the worker processes sharing a connection. One place where we create a network connection very early during process startup is to check the APM Server version. This is a fairly new feature, released with version 6.4.0 of the agent on September 3. Did you see these exceptions any earlier than that?
Yes, we already saw them in 2020 already.

Resolving patch:
diff --git a/elasticapm/transport/base.py b/elasticapm/transport/base.py
--- a/elasticapm/transport/base.py
+++ b/elasticapm/transport/base.py
@@ -268,6 +268,7 @@ def _flush(self, buffer):
     def start_thread(self, pid=None):
         super(Transport, self).start_thread(pid=pid)
         if (not self._thread or self.pid != self._thread.pid) and not self._closed:
+            self.handle_fork()
             try:
                 self._thread = threading.Thread(target=self._process_queue, name="eapm event processor thread")
                 self._thread.daemon = True
@@ -321,6 +322,10 @@ def handle_transport_fail(self, exception=None, **kwargs):
         logger.error("Failed to submit message: %r", message, exc_info=getattr(exception, "print_trace", True))
         self.state.set_fail()
 
+    def handle_fork(self) -> None:
+        """Helper method to run code after a fork has been detected"""
+        pass
+
 
 # left for backwards compatibility
 AsyncTransport = Transport
diff --git a/elasticapm/transport/http.py b/elasticapm/transport/http.py
--- a/elasticapm/transport/http.py
+++ b/elasticapm/transport/http.py
@@ -52,24 +52,21 @@
 
 
 class Transport(HTTPTransportBase):
-    def __init__(self, url, *args, **kwargs):
+    def __init__(self, url: str, *args, **kwargs) -> None:
         super(Transport, self).__init__(url, *args, **kwargs)
-        url_parts = compat.urlparse.urlparse(url)
         pool_kwargs = {"cert_reqs": "CERT_REQUIRED", "ca_certs": self.ca_certs, "block": True}
-        if self._server_cert and url_parts.scheme != "http":
-            pool_kwargs.update(
-                {"assert_fingerprint": self.cert_fingerprint, "assert_hostname": False, "cert_reqs": ssl.CERT_NONE}
-            )
-            del pool_kwargs["ca_certs"]
-        elif not self._verify_server_cert and url_parts.scheme != "http":
-            pool_kwargs["cert_reqs"] = ssl.CERT_NONE
-            pool_kwargs["assert_hostname"] = False
-        proxies = compat.getproxies_environment()
-        proxy_url = proxies.get("https", proxies.get("http", None))
-        if proxy_url and not compat.proxy_bypass_environment(url_parts.netloc):
-            self.http = urllib3.ProxyManager(proxy_url, **pool_kwargs)
-        else:
-            self.http = urllib3.PoolManager(**pool_kwargs)
+        if url.startswith("https"):
+            if self._server_cert:
+                pool_kwargs.update(
+                    {"assert_fingerprint": self.cert_fingerprint, "assert_hostname": False, "cert_reqs": ssl.CERT_NONE}
+                )
+                del pool_kwargs["ca_certs"]
+            elif not self._verify_server_cert:
+                pool_kwargs["cert_reqs"] = ssl.CERT_NONE
+                pool_kwargs["assert_hostname"] = False
+        self._pool_kwargs = pool_kwargs
+        self._http = None
+        self._url = url
 
     def send(self, data):
         response = None
@@ -113,6 +110,22 @@ def send(self, data):
             if response:
                 response.close()
 
+    @property
+    def http(self) -> urllib3.PoolManager:
+        if not self._http:
+            url_parts = compat.urlparse.urlparse(self._url)
+            proxies = compat.getproxies_environment()
+            proxy_url = proxies.get("https", proxies.get("http", None))
+            if proxy_url and not compat.proxy_bypass_environment(url_parts.netloc):
+                self._http = urllib3.ProxyManager(proxy_url, **self._pool_kwargs)
+            else:
+                self._http = urllib3.PoolManager(**self._pool_kwargs)
+        return self._http
+
+    def handle_fork(self) -> None:
+        # reset http pool to avoid sharing connections with the parent process
+        self._http = None
+
     def get_config(self, current_version=None, keys=None):
         """
         Gets configuration from a remote APM Server


Test patch: