Repository: DataDog/dd-trace-py

Issue:
No trace for elasticsearch7
My team's app has to work with multiple versions of elasticsearch, so we use the version-specific libraries. I saw in the [0.46.0 release](https://github.com/DataDog/dd-trace-py/releases/tag/v0.46.0) that y'all added support for `elasticsearch7` so I immediately upgraded and gave it a shot. Unfortunately, I'm not seeing any spans for elasticsearch in the trace.

### Which version of dd-trace-py are you using?
0.46.0

### Which version of the libraries are you using?
Here are the ES libs we're using:
```
elasticsearch              1.9.0
elasticsearch-dsl          0.0.11
elasticsearch7             7.11.0
elasticsearch7-dsl         7.2.1
```

### Tracer setup
This is at the top of our WSGI:

If it matters, we're running a django 1.8

```python
from ddtrace import config, patch_all

patch_all(httplib=True)
# header names are not case sensitive here
config.trace_headers([
    'cache-control',
    'expires',
    'host',
    'referer',
    'server',
    'user-agent',
])
config.django.http.trace_query_string = True
```



Discussion:
Thanks for the report @jprestel-rue! Strange as support should have been included in 0.46 and I see it included in the tagged commit.

It's now showing up in the 0.47 release notes and not the 0.46, super strange 🤔. We'll take a look!

In the meantime could you try 0.47?
Trying with v0.47.0 now. I'll let you know how it goes!
No luck. Still not getting spans for ES with v0.47.0 :(
Thanks for the super quick follow-up. We'll dig into it
I set `DD_TRACE_DEBUG=true`. Found this in the output:
```json
{
    "date": "2021-03-03T20:45:09.290016",
    "os_name": "Linux",
    "os_version": "4.14.193-149.317.amzn2.x86_64",
    "is_64_bit": true,
    "architecture": "64bit",
    "vm": "CPython",
    "version": "0.47.0",
    "lang": "python",
    "lang_version": "3.6.10",
    "pip_version": "21.0.1",
    "in_virtual_env": false,
    "agent_url": "http://<redacted>:8126",
    "agent_error": null,
    "env": "orca",
    "is_global_tracer": true,
    "enabled_env_setting": null,
    "tracer_enabled": true,
    "sampler_type": "DatadogSampler",
    "priority_sampler_type": "RateByServiceSampler",
    "service": "product-indexer",
    "debug": true,
    "enabled_cli": false,
    "analytics_enabled": false,
    "log_injection_enabled": false,
    "health_metrics_enabled": false,
    "dd_version": "",
    "priority_sampling_enabled": true,
    "global_tags": "",
    "tracer_tags": "",
    "integrations": {
        "asyncio": "N/A",
        "boto": "N/A",
        "botocore": {
            "enabled": true,
            "instrumented": false,
            "module_available": true,
            "module_version": "1.11.9",
            "module_imported": true,
            "config": "N/A"
        },
        "bottle": "N/A",
        "cassandra": "N/A",
        "celery": {
            "enabled": true,
            "instrumented": false,
            "module_available": true,
            "module_version": "3.1.24",
            "module_imported": false,
            "config": "N/A"
        },
        "consul": "N/A",
        "django": "N/A",
        "elasticsearch": {
            "enabled": true,
            "instrumented": false,
            "module_available": true,
            "module_version": "7.6.0",
            "module_imported": false,
            "config": "N/A"
        },
        "algoliasearch": "N/A",
        "futures": "N/A",
        "grpc": "N/A",
        "mongoengine": "N/A",
        "mysql": "N/A",
        "mysqldb": "N/A",
        "pymysql": "N/A",
        "psycopg": "N/A",
        "pylibmc": "N/A",
        "pymemcache": "N/A",
        "pymongo": "N/A",
        "redis": {
            "enabled": true,
            "instrumented": false,
            "module_available": true,
            "module_version": "2.10.5",
            "module_imported": true,
            "config": "N/A"
        },
        "rediscluster": "N/A",
        "requests": {
            "enabled": true,
            "instrumented": false,
            "module_available": true,
            "module_version": "2.7.0",
            "module_imported": true,
            "config": "N/A"
        },
        "sanic": "N/A",
        "sqlalchemy": "N/A",
        "sqlite3": "N/A",
        "aiohttp": "N/A",
        "aiopg": "N/A",
        "aiobotocore": "N/A",
        "httplib": "N/A",
        "vertica": "N/A",
        "molten": "N/A",
        "jinja2": "N/A",
        "mako": "N/A",
        "flask": "N/A",
        "kombu": {
            "enabled": false,
            "instrumented": false,
            "module_available": true,
            "module_version": "3.0.37",
            "module_imported": false,
            "config": null
        },
        "starlette": "N/A",
        "falcon": "N/A",
        "pylons": "N/A",
        "pyramid": "N/A",
        "logging": "N/A",
        "pynamodb": "N/A",
        "pyodbc": "N/A",
        "fastapi": "N/A"
    }
}
```

and this:

```
INFO:ddtrace.monkey:patched 13/35 modules (aiohttp,asyncio,botocore,celery,django,elasticsearch,flask,httplib,mysqldb,pynamodb,redis,requests,sqlite3)
```
If it's helpful in any way, we used to have this sort of monkey patch to get partial tracing for es7 library. I removed it when upgrading to 0.47.0:

```python
from importlib import import_module

from ddtrace.contrib.elasticsearch.patch import _patch

# Patch the ES7 module
# This logic is mimiced from the below snippet
# https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/contrib/elasticsearch/patch.py#L15-L27
_patch(import_module('elasticsearch7'))
```

With that hack, we were able to get spans for Elasticsearch show up in traces. 
This graph from APM shows how the Elasticsearch visibility disappeared after removing that patch and using 0.47.0's built-in support instead:
![image](https://user-images.githubusercontent.com/17439030/109871776-89188e00-7c39-11eb-95a0-616bf53bd788.png)

Hey @Kyle-Verhoog just wanted to check in to see if y'all had learned anything or if there's any additional info I can help provide.
@jprestel-rue I'm going to take a look at it today 🙂.

Resolving patch:
diff --git a/ddtrace/contrib/elasticsearch/__init__.py b/ddtrace/contrib/elasticsearch/__init__.py
--- a/ddtrace/contrib/elasticsearch/__init__.py
+++ b/ddtrace/contrib/elasticsearch/__init__.py
@@ -19,22 +19,7 @@
     Pin.override(es.transport, service='elasticsearch-videos')
     es.indices.create(index='videos', ignore=400)
 """
-from ...utils.importlib import require_modules
+from .patch import patch
 
 
-# DEV: We only require one of these modules to be available
-required_modules = [
-    "elasticsearch",
-    "elasticsearch1",
-    "elasticsearch2",
-    "elasticsearch5",
-    "elasticsearch6",
-    "elasticsearch7",
-]
-
-with require_modules(required_modules) as missing_modules:
-    # We were able to find at least one of the required modules
-    if set(missing_modules) != set(required_modules):
-        from .patch import patch
-
-        __all__ = ["patch"]
+__all__ = ["patch"]
diff --git a/ddtrace/monkey.py b/ddtrace/monkey.py
--- a/ddtrace/monkey.py
+++ b/ddtrace/monkey.py
@@ -91,7 +91,13 @@
     "gevent": ("gevent",),
     "requests": ("requests",),
     "botocore": ("botocore",),
-    "elasticsearch": ("elasticsearch",),
+    "elasticsearch": (
+        "elasticsearch",
+        "elasticsearch2",
+        "elasticsearch5",
+        "elasticsearch6",
+        "elasticsearch7",
+    ),
     "pynamodb": ("pynamodb",),
 }
 
@@ -113,8 +119,14 @@ def _on_import_factory(module, raise_errors=True):
     def on_import(hook):
         # Import and patch module
         path = "ddtrace.contrib.%s" % module
-        imported_module = importlib.import_module(path)
-        imported_module.patch()
+        try:
+            imported_module = importlib.import_module(path)
+        except ImportError:
+            if raise_errors:
+                raise
+            log.error("failed to import ddtrace module %r when patching on import", path, exc_info=True)
+        else:
+            imported_module.patch()
 
     return on_import
 
@@ -161,18 +173,19 @@ def patch(raise_errors=True, **patch_modules):
     modules = [m for (m, should_patch) in patch_modules.items() if should_patch]
     for module in modules:
         if module in _PATCH_ON_IMPORT:
-            # If the module has already been imported then patch immediately
-            if module in sys.modules:
-                patch_module(module, raise_errors=raise_errors)
-
-            # Otherwise, add a hook to patch when it is imported for the first time
-            else:
-                # Use factory to create handler to close over `module` and `raise_errors` values from this loop
-                when_imported(module)(_on_import_factory(module, raise_errors))
-
-                # manually add module to patched modules
-                with _LOCK:
-                    _PATCHED_MODULES.add(module)
+            modules_to_poi = _PATCH_ON_IMPORT[module]
+            for m in modules_to_poi:
+                # If the module has already been imported then patch immediately
+                if m in sys.modules:
+                    patch_module(m, raise_errors=raise_errors)
+                # Otherwise, add a hook to patch when it is imported for the first time
+                else:
+                    # Use factory to create handler to close over `module` and `raise_errors` values from this loop
+                    when_imported(m)(_on_import_factory(module, raise_errors))
+
+            # manually add module to patched modules
+            with _LOCK:
+                _PATCHED_MODULES.add(module)
         else:
             patch_module(module, raise_errors=raise_errors)
 
diff --git a/riotfile.py b/riotfile.py
--- a/riotfile.py
+++ b/riotfile.py
@@ -270,7 +270,7 @@ def select_pys(min_version=min(SUPPORTED_PYTHON_VERSIONS), max_version=max(SUPPO
         ),
         Venv(
             name="elasticsearch",
-            command="pytest {cmdargs} tests/contrib/elasticsearch",
+            command="pytest {cmdargs} tests/contrib/elasticsearch/test_elasticsearch.py",
             venvs=[
                 Venv(
                     pys=select_pys(max_version=3.8),
@@ -313,6 +313,22 @@ def select_pys(min_version=min(SUPPORTED_PYTHON_VERSIONS), max_version=max(SUPPO
                 Venv(pys=select_pys(), pkgs={"elasticsearch7": ["~=7.6.0", "~=7.8.0", "~=7.10.0", latest]}),
             ],
         ),
+        Venv(
+            name="elasticsearch-multi",
+            command="pytest {cmdargs} tests/contrib/elasticsearch/test_elasticsearch_multi.py",
+            venvs=[
+                Venv(
+                    pys=select_pys(),
+                    pkgs={
+                        "elasticsearch": ["~=1.6.0"],
+                        "elasticsearch2": [latest],
+                        "elasticsearch5": [latest],
+                        "elasticsearch6": [latest],
+                        "elasticsearch7": [latest],
+                    },
+                ),
+            ],
+        ),
         Venv(
             name="flask",
             command="pytest {cmdargs} tests/contrib/flask",


Test patch:
diff --git a/tests/contrib/elasticsearch/test.py b/tests/contrib/elasticsearch/test_elasticsearch.py
similarity index 100%
rename from tests/contrib/elasticsearch/test.py
rename to tests/contrib/elasticsearch/test_elasticsearch.py
diff --git a/tests/contrib/elasticsearch/test_elasticsearch_multi.py b/tests/contrib/elasticsearch/test_elasticsearch_multi.py
new file mode 100644
--- /dev/null
+++ b/tests/contrib/elasticsearch/test_elasticsearch_multi.py
@@ -0,0 +1,72 @@
+import os
+import subprocess
+import sys
+
+from ddtrace.vendor import six
+from tests import snapshot
+
+
+code = """
+import datetime
+import os
+
+import %s as elasticsearch
+
+ELASTICSEARCH_CONFIG = {"port": int(os.getenv("TEST_ELASTICSEARCH_PORT", 9200)),}
+ES_INDEX = "ddtrace_index"
+ES_TYPE = "ddtrace_type"
+mapping = {"mapping": {"properties": {"created": {"type": "date", "format": "yyyy-MM-dd"}}}}
+es = elasticsearch.Elasticsearch(port=ELASTICSEARCH_CONFIG["port"])
+es.indices.create(index=ES_INDEX, ignore=400, body=mapping)
+
+args = {"index": ES_INDEX, "doc_type": ES_TYPE}
+es.indices.delete(index=ES_INDEX, ignore=[400, 404])
+"""
+
+
+def do_test(tmpdir, es_version):
+    f = tmpdir.join("test.py")
+    f.write(code % es_version)
+    env = os.environ.copy()
+    # ddtrace-run patches sqlite3 which is used by coverage to store coverage
+    # results. This generates sqlite3 spans during the test run which interfere
+    # with the snapshot. So disable sqlite3.
+    env.update({"DD_TRACE_SQLITE3_ENABLED": "false"})
+    p = subprocess.Popen(
+        ["ddtrace-run", sys.executable, "test.py"],
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        cwd=str(tmpdir),
+        env=env,
+    )
+    p.wait()
+    stderr = p.stderr.read()
+    stdout = p.stdout.read()
+    assert stderr == six.b(""), stderr
+    assert stdout == six.b(""), stdout
+    assert p.returncode == 0
+
+
+@snapshot(async_mode=False)
+def test_elasticsearch(tmpdir):
+    do_test(tmpdir, "elasticsearch")
+
+
+@snapshot(async_mode=False)
+def test_elasticsearch2(tmpdir):
+    do_test(tmpdir, "elasticsearch2")
+
+
+@snapshot(async_mode=False)
+def test_elasticsearch5(tmpdir):
+    do_test(tmpdir, "elasticsearch5")
+
+
+@snapshot(async_mode=False)
+def test_elasticsearch6(tmpdir):
+    do_test(tmpdir, "elasticsearch6")
+
+
+@snapshot(async_mode=False)
+def test_elasticsearch7(tmpdir):
+    do_test(tmpdir, "elasticsearch7")
diff --git a/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch.snap b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch.snap
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch.snap
@@ -0,0 +1,40 @@
+[[{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "PUT /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 0
+   "parent_id" nil
+   "start" 1615404339005889000
+   "duration" 94071000
+   "meta" {"runtime-id" "f64590d3c3cf4b578d03f6de8cdf1b5c"
+           "elasticsearch.method" "PUT"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=400"
+           "http.status_code" "200"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51885
+              "_dd.tracer_kr" 1.0}}]
+ [{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "DELETE /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 1
+   "parent_id" nil
+   "start" 1615404339100245000
+   "duration" 46499000
+   "meta" {"runtime-id" "f64590d3c3cf4b578d03f6de8cdf1b5c"
+           "elasticsearch.method" "DELETE"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=%5B400%2C+404%5D"
+           "http.status_code" "200"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51885
+              "_dd.tracer_kr" 1.0}}]]
diff --git a/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch2.snap b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch2.snap
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch2.snap
@@ -0,0 +1,38 @@
+[[{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "PUT /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 0
+   "parent_id" nil
+   "start" 1615404339946917000
+   "duration" 54144000
+   "meta" {"runtime-id" "b57310def5e6453589e773a8c58561be"
+           "elasticsearch.method" "PUT"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=400"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51889
+              "_dd.tracer_kr" 1.0}}]
+ [{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "DELETE /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 1
+   "parent_id" nil
+   "start" 1615404340001313000
+   "duration" 36811000
+   "meta" {"runtime-id" "b57310def5e6453589e773a8c58561be"
+           "elasticsearch.method" "DELETE"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=%5B400%2C+404%5D"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51889
+              "_dd.tracer_kr" 1.0}}]]
diff --git a/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch5.snap b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch5.snap
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch5.snap
@@ -0,0 +1,38 @@
+[[{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "PUT /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 0
+   "parent_id" nil
+   "start" 1615404340842015000
+   "duration" 29740000
+   "meta" {"runtime-id" "87d21b7e83ce4c79853c35ed9cb8a6bb"
+           "elasticsearch.method" "PUT"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=400"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51892
+              "_dd.tracer_kr" 1.0}}]
+ [{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "DELETE /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 1
+   "parent_id" nil
+   "start" 1615404340872051000
+   "duration" 34415000
+   "meta" {"runtime-id" "87d21b7e83ce4c79853c35ed9cb8a6bb"
+           "elasticsearch.method" "DELETE"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=%5B400%2C+404%5D"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51892
+              "_dd.tracer_kr" 1.0}}]]
diff --git a/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch6.snap b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch6.snap
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch6.snap
@@ -0,0 +1,38 @@
+[[{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "PUT /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 0
+   "parent_id" nil
+   "start" 1615404365434553000
+   "duration" 32898000
+   "meta" {"runtime-id" "0a66135faff745d78feb7c40c4f70869"
+           "elasticsearch.method" "PUT"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=400"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51995
+              "_dd.tracer_kr" 1.0}}]
+ [{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "DELETE /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 1
+   "parent_id" nil
+   "start" 1615404365467709000
+   "duration" 27295000
+   "meta" {"runtime-id" "0a66135faff745d78feb7c40c4f70869"
+           "elasticsearch.method" "DELETE"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=%5B400%2C+404%5D"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51995
+              "_dd.tracer_kr" 1.0}}]]
diff --git a/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch7.snap b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch7.snap
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.elasticsearch.test_elasticsearch_multi.test_elasticsearch7.snap
@@ -0,0 +1,38 @@
+[[{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "PUT /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 0
+   "parent_id" nil
+   "start" 1615404342493231000
+   "duration" 44487000
+   "meta" {"runtime-id" "33c55b5f693d44d481205163652f4a4c"
+           "elasticsearch.method" "PUT"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=400"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51898
+              "_dd.tracer_kr" 1.0}}]
+ [{"name" "elasticsearch.query"
+   "service" "elasticsearch"
+   "resource" "DELETE /ddtrace_index"
+   "type" "elasticsearch"
+   "error" 0
+   "span_id" 0
+   "trace_id" 1
+   "parent_id" nil
+   "start" 1615404342538001000
+   "duration" 39621000
+   "meta" {"runtime-id" "33c55b5f693d44d481205163652f4a4c"
+           "elasticsearch.method" "DELETE"
+           "elasticsearch.url" "/ddtrace_index"
+           "elasticsearch.params" "ignore=%5B400%2C+404%5D"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 51898
+              "_dd.tracer_kr" 1.0}}]]