Repository: DataDog/dd-trace-py

Issue:
[boto] [botocore] importing boto/botocore before gevent patching raises exception
Reported in https://github.com/DataDog/dd-trace-py/issues/506#issuecomment-431692318

This is a similar problem as #632 


To reproduce:

```python
# app.py
import gevent.monkey
gevent.monkey.patch_all()
```

```
$ ddtrace-run python app.py
app.py:2: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['botocore.httpsession (/Users/brett.langdon/.env/issue-662/lib/python3.7/site-packages/botocore/httpsession.py)', 'botocore.vendored.requests.packages.urllib3.util.ssl_ (/Users/brett.langdon/.env/issue-662/lib/python3.7/site-packages/botocore/vendored/requests/packages/urllib3/util/ssl_.py)', 'botocore.vendored.requests.packages.urllib3.util (/Users/brett.langdon/.env/issue-662/lib/python3.7/site-packages/botocore/vendored/requests/packages/urllib3/util/__init__.py)', 'urllib3.util (/Users/brett.langdon/.env/issue-662/lib/python3.7/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/Users/brett.langdon/.env/issue-662/lib/python3.7/site-packages/urllib3/util/ssl_.py)'].
  gevent.monkey.patch_all()
```



Discussion:
Hey @brettlangdon! 

If you're looking for a solution to this. I found it's because botocore while being patched gets imported, it imports some of it's own dependencies, including urllib.

The solution is a little bit messy. So with `ddtrace-run gunicorn -k gevent` I found the solution was to disable botocore, and anything else that imports urllib like so:

```
DATADOG_PATCH_MODULES=botocore:false,elasticsearch:false,requests:false ddtrace-run gunicorn -k gevent app
```
Then explicitly patch them again in your app.py

```
import ddtrace
ddtrace.patch(
    requests=True, botocore=True, elasticsearch=True
)
```

Also make sure you're using gevent 1.3.4 or greater to silence the now needless monkeypatch warning.

:)
We can address this by enabling patching on import for more of our integrations, particularly those that have `urllib3` as a dependency. Once that's completed, this workaround will not be necessary.
We're on our first day of evaluating APM; this bug kills the application, so in our case it's a blocker.
Any update on this?

Hi DD people,

I'm also experiencing this issue with:
```
gunicorn==20.0.4
gevent==20.6.2
datadog==0.38.0
ddtrace==0.40.0
```
Executing like: 
```
ddtrace-run gunicorn --worker-class=gevent ...
```

So to summarise, this issue is still open, with only [this solution](https://github.com/DataDog/dd-trace-py/issues/662#issuecomment-482080556) (that btw fixes my problem also).

The major problem here is this issue not being referred anywhere in your setup docs: http://pypi.datadoghq.com/trace/docs/async_integrations.html#gevent or https://docs.datadoghq.com/tracing/setup/python/ .

So if it wont be fixed anytime soon (already more than a year 🙃), I think you should document it properly, as it's a blocker for us (all gevent and boto/requests users). 

@brettlangdon 🙏 

Resolving patch:
diff --git a/ddtrace/contrib/gevent/__init__.py b/ddtrace/contrib/gevent/__init__.py
--- a/ddtrace/contrib/gevent/__init__.py
+++ b/ddtrace/contrib/gevent/__init__.py
@@ -1,25 +1,40 @@
 """
-To trace a request in a ``gevent`` environment, configure the tracer to use the greenlet
-context provider, rather than the default one that relies on a thread-local storaging.
+The gevent integration adds support for tracing across greenlets.
 
-This allows the tracer to pick up a transaction exactly where it left off as greenlets
-yield the context to another one.
+The integration patches the gevent internals to add context management logic.
+It also configures the global tracer instance to use a gevent context
+provider to utilize the context management logic.
 
-The simplest way to trace a ``gevent`` application is to configure the tracer and
-patch ``gevent`` **before importing** the library::
+If custom tracer instances are being used in a gevent application, then
+configure it with::
 
-    # patch before importing gevent
-    from ddtrace import patch, tracer
+    from ddtrace.contrib.gevent import context_provider
+
+    tracer.configure(context_provider=context_provider)
+
+
+Enabling
+~~~~~~~~
+
+The integration is enabled automatically when using
+:ref:`ddtrace-run<ddtracerun>` or :ref:`patch_all()<patch_all>`.
+
+Or use :ref:`patch()<patch>` to manually enable the integration::
+
+    from ddtrace import patch
     patch(gevent=True)
 
-    # use gevent as usual with or without the monkey module
-    from gevent import monkey; monkey.patch_thread()
+
+**Note:** these calls need to be performed before calling the gevent patching.
+
+Example of the context propagation::
 
     def my_parent_function():
         with tracer.trace("web.request") as span:
             span.service = "web"
             gevent.spawn(worker_function)
 
+
     def worker_function():
         # then trace its child
         with tracer.trace("greenlet.call") as span:
@@ -32,7 +47,7 @@ def worker_function():
 from ...utils.importlib import require_modules
 
 
-required_modules = ['gevent']
+required_modules = ["gevent"]
 
 with require_modules(required_modules) as missing_modules:
     if not missing_modules:
@@ -42,7 +57,7 @@ def worker_function():
         context_provider = GeventContextProvider()
 
         __all__ = [
-            'patch',
-            'unpatch',
-            'context_provider',
+            "patch",
+            "unpatch",
+            "context_provider",
         ]
diff --git a/ddtrace/monkey.py b/ddtrace/monkey.py
--- a/ddtrace/monkey.py
+++ b/ddtrace/monkey.py
@@ -75,10 +75,14 @@
 # DEV: This ensures we do not patch a module until it is needed
 # DEV: <contrib name> => <list of module names that trigger a patch>
 _PATCH_ON_IMPORT = {
+    "aiohttp": ("aiohttp",),
+    "aiobotocore": ("aiobotocore",),
     "celery": ("celery",),
     "flask": ("flask, "),
     "gevent": ("gevent",),
     "requests": ("requests",),
+    "botocore": ("botocore",),
+    "elasticsearch": ("elasticsearch",),
 }
 
 


Test patch:
diff --git a/tests/contrib/gevent/monkeypatch.py b/tests/contrib/gevent/monkeypatch.py
new file mode 100644
--- /dev/null
+++ b/tests/contrib/gevent/monkeypatch.py
@@ -0,0 +1,3 @@
+import gevent.monkey
+gevent.monkey.patch_all()
+print("Test success")
diff --git a/tests/contrib/gevent/test_tracer.py b/tests/contrib/gevent/test_tracer.py
--- a/tests/contrib/gevent/test_tracer.py
+++ b/tests/contrib/gevent/test_tracer.py
@@ -1,21 +1,24 @@
+import subprocess
+
 import gevent
 import gevent.pool
 import ddtrace
+from ddtrace.compat import PY3
 
 from ddtrace.constants import SAMPLING_PRIORITY_KEY
 from ddtrace.context import Context
 from ddtrace.contrib.gevent import patch, unpatch
 from ddtrace.ext.priority import USER_KEEP
 
-from unittest import TestCase
 from opentracing.scope_managers.gevent import GeventScopeManager
 from tests.opentracer.utils import init_tracer
 from tests.tracer.test_tracer import get_dummy_tracer
+from tests import TracerTestCase
 
 from .utils import silence_errors
 
 
-class TestGeventTracer(TestCase):
+class TestGeventTracer(TracerTestCase):
     """
     Ensures that greenlets are properly traced when using
     the default Tracer.
@@ -436,3 +439,33 @@ def green_2():
 
         spans = self.tracer.writer.pop()
         self._assert_spawn_multiple_greenlets(spans)
+
+    def test_ddtracerun(self):
+        """
+        Regression test case for the following issue.
+
+        ddtrace-run imports all available modules in order to patch them.
+        However, gevent depends on the ssl module not being imported when it
+        goes to monkeypatch. Modules that import ssl include botocore, requests
+        and elasticsearch.
+        """
+
+        # Ensure modules are installed
+        if PY3:
+            import aiohttp  # noqa
+            import aiobotocore  # noqa
+        import botocore  # noqa
+        import requests  # noqa
+        import elasticsearch  # noqa
+
+        p = subprocess.Popen(
+            ["ddtrace-run", "python", "tests/contrib/gevent/monkeypatch.py"],
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        p.wait()
+        stdout, stderr = p.stdout.read(), p.stderr.read()
+        assert p.returncode == 0, stderr
+        assert b"Test success" in stdout
+        assert b"RecursionError" not in stderr