Repository: DataDog/dd-trace-py

Issue:
Lost visibility into bulk Elasticsearch operations with 0.48.0
With upgrade to ddtrace 0.48.0, we appear to have lost visibility into Elasticsearch bulk operations.

Here's the view in APM for our indexer application that writes to ES. We seem to have lost all visibility. The first screenshot shows that APM istelf is still working, while the second screenshot makes it clear that we lost all visibility into Elasticsearch.

![image](https://user-images.githubusercontent.com/17439030/113631013-cdcb8680-9636-11eb-8699-77dc8d51a54a.png)

![image](https://user-images.githubusercontent.com/17439030/113630896-9fe64200-9636-11eb-90f7-ccdf93a0cbe7.png)

Within a trace, I can see the `http.client.request` to ES, but again, no visibility into ES as a service.


### Which version of dd-trace-py are you using?
0.48.0

### Which version of pip are you using?
18.1

### Which version of the libraries are you using?
```
elasticsearch7==7.12.0
elasticsearch7-dsl==7.2.1
```

### How can we reproduce your problem?
Setup APM on an application that writes to ES with bulk operations.

### What is the result that you get?
No span for the bulk write to ES

### What is the result that you expected?
You should get a span for the bulk write to ES


Discussion:
Hi there, @jprestel-rue. I was hoping to get some more information from you if that's OK. I was trying to reproduce the problem, but from my end I can see the `elasticsearch` spans when making bulk requests. This is a simple script that I have tried
~~~ python
import datetime
import elasticsearch7 as elasticsearch
import elasticsearch7.helpers as helpers

ES_INDEX = "ddtrace_index"
ES_TYPE = "ddtrace_type"

es = elasticsearch.Elasticsearch(port=9200)
mapping = {"mapping": {"properties": {"created": {"type": "date", "format": "yyyy-MM-dd"}}}}
es.indices.create(index=ES_INDEX, ignore=400, body=mapping)

actions = [
    {
        "_index": ES_INDEX,
        "_type": ES_TYPE,
        "_id": j,
        "_source": {"any": "data" + str(j), "timestamp": datetime.datetime.now()},
    }
    for j in range(10)
]
helpers.bulk(es, actions)
~~~

and these are some of the spans that are being recorded

<img width="750" alt="Screenshot 2021-05-04 at 15 04 08" src="https://user-images.githubusercontent.com/20231758/117015849-06b85300-acea-11eb-82d2-24c6d0862f45.png" />

Does `ddtrace-run` show any errors in the logs when running with `DD_TRACE_DEBUG=1`? Also, just to confirm, is it the case that other `elasticsearch` spans are present, just the bulk ones are missing?
I enabled both `DD_TRACE_DEBUG` and `DD_TRACE_STARTUP_LOGS`.

Here are some interesting snippets:

```
INFO:ddtrace.tracer:- DATADOG TRACER CONFIGURATION - {"date": "2021-05-05T00:05:10.334035", "os_name": "Linux", "os_version": "4.14.193-149.317.amzn2.x86_64", "is_64_bit": true, "architecture": "64bit", "vm": "CPython", "version": "0.48.2", "lang": "python", "lang_version": "3.6.10", "pip_version": "21.1.1", "in_virtual_env": false, "agent_url": "http://172.17.0.1:8126", "agent_error": null, "env": "orca", "is_global_tracer": true, "enabled_env_setting": null, "tracer_enabled": true, "sampler_type": "DatadogSampler", "priority_sampler_type": "RateByServiceSampler", "service": "product-indexer", "debug": true, "enabled_cli": false, "analytics_enabled": false, "log_injection_enabled": false, "health_metrics_enabled": false, "dd_version": "", "priority_sampling_enabled": true, "global_tags": "", "tracer_tags": "", "integrations": {"asyncio": "N/A", "boto": "N/A", "botocore": {"enabled": true, "instrumented": false, "module_available": true, "module_version": "1.11.9", "module_imported": true, "config": "N/A"}, "bottle": "N/A", "cassandra": "N/A", "celery": {"enabled": true, "instrumented": false, "module_available": true, "module_version": "3.1.24", "module_imported": false, "config": "N/A"}, "consul": "N/A", "django": "N/A", "elasticsearch": {"enabled": true, "instrumented": false, "module_available": true, "module_version": "7.6.0", "module_imported": false, "config": "N/A"}, "algoliasearch": "N/A", "futures": "N/A", "grpc": "N/A", "mongoengine": "N/A", "mysql": "N/A", "mysqldb": "N/A", "pymysql": "N/A", "psycopg": "N/A", "pylibmc": "N/A", "pymemcache": "N/A", "pymongo": "N/A", "redis": {"enabled": true, "instrumented": false, "module_available": true, "module_version": "2.10.5", "module_imported": true, "config": "N/A"}, "rediscluster": "N/A", "requests": {"enabled": true, "instrumented": false, "module_available": true, "module_version": "2.7.0", "module_imported": true, "config": "N/A"}, "sanic": "N/A", "sqlalchemy": "N/A", "sqlite3": "N/A", "aiohttp": "N/A", "aiopg": "N/A", "aiobotocore": "N/A", "httplib": "N/A", "urllib3": {"enabled": false, "instrumented": false, "module_available": true, "module_version": "1.23", "module_imported": true, "config": null}, "vertica": "N/A", "molten": "N/A", "jinja2": "N/A", "mako": "N/A", "flask": "N/A", "kombu": {"enabled": false, "instrumented": false, "module_available": true, "module_version": "3.0.37", "module_imported": false, "config": null}, "starlette": "N/A", "falcon": "N/A", "pylons": "N/A", "pyramid": "N/A", "logging": "N/A", "pynamodb": "N/A", "pyodbc": "N/A", "fastapi": "N/A", "dogpile_cache": "N/A"}}
```

```
DEBUG:ddtrace.monkey:failed to patch elasticsearch7
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/ddtrace/monkey.py", line 242, in _patch_module
imported_module = importlib.import_module(path)
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'ddtrace.contrib.elasticsearch7'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/ddtrace/monkey.py", line 209, in patch_module
return _patch_module(module)
File "/usr/local/lib/python3.6/site-packages/ddtrace/monkey.py", line 245, in _patch_module
raise PatchException("integration '%s' not available" % path)
ddtrace.monkey.PatchException: integration 'ddtrace.contrib.elasticsearch7' not available
```

Let me know if you need to see anything else!
That's great, @jprestel-rue, thanks for the logs. I was able to reproduce the same traceback using

~~~ python
patch(elasticsearch7=True)
~~~

If you have a similar line in your code, may I ask you to either omit it (`elasticsearch` is patched on import by default when using `ddtrace-run` or `patch_all`), or change `elasticsearch7=True` to `elasticsearch=True`.
Our patch setup looks like this:

```python
from ddtrace import patch_all, tracer

...

tracer.configure(hostname='172.17.0.1', port=8126)
patch_all(httplib=True)
```


I couldn't find any instance where we set `patch(elasticsearch7=True)`
Hmm, that's weird. With that I cannot reproduce the problem. Based on the traceback provided earlier, the only ways the problem could be occurring are if there are calls like the already mentioned `patch(elasticsearch7=True)`, or perhaps `patch_module("elasticsearch7")`. The library can handle different versions of elasticsearch, so manual activations should reference the generic `elasticsearch` rather than, e.g., `elasticsearch7`. This explains the log line
~~~
ddtrace.monkey.PatchException: integration 'ddtrace.contrib.elasticsearch7' not available
~~~
We _**used to**_ have a hack in place that looked like this:

```python
from ddtrace import patch_all, tracer

...

from importlib import import_module
from ddtrace.contrib.elasticsearch.patch import _patch

tracer.configure(hostname='172.17.0.1', port=8126)
# Patch the ES7 module on app ready
# this logic is mimiced from the below snippet
# https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/contrib/elasticsearch/patch.py#L15-L27
_patch(import_module('elasticsearch7'))
```

This was before y'all added proper support. But it's definitely removed from the codebase, and I can't find any evidence that there's residual .pyc or anything.

There's just no evidence (that I can find) of a call to explicitly patch the es7 module anywhere in our code 😞 


Could this possibly be a sequencing thing?

For context, what we're instrumenting is a Django (v1.8) management command. The patching is configured inside our `AppConfig` for the application. I don't know the exact sequence that Django is loading things before the `AppConfig` is processed.

In contrast, I haven't seen any issues with our web app, which is built on the same codebase, more or less. The Django web app is running via Apache mod_wsgi, and the patching is done immediately inside the `wsgi` file.

So is there a chance that the difference in order of operations could cause this? </ longshot> 😬 
> So is there a chance that the difference in order of operations could cause this? </ longshot>

Indeed, this could also be caused by the order in which things are executed. I have just been able to reproduce the problem by importing `elasticsearch7` _before_ patching. That is,

~~~ python
import datetime

import elasticsearch7 as elasticsearch

from ddtrace import patch_all
from ddtrace import tracer


tracer.configure(port=8126)
patch_all(httplib=True)
~~~
also reproduces the problem. However,

~~~ python
import datetime

from ddtrace import patch_all
from ddtrace import tracer


tracer.configure(port=8126)
patch_all(httplib=True)

import elasticsearch7 as elasticsearch
~~~
i.e. importing _after_ calling `patch_all` should work as expected. Are there any chances that you could try changing the order by which things happen while I see if we can provide a fix for this? Thanks for all the help and the patience thus far!

Resolving patch:
diff --git a/ddtrace/monkey.py b/ddtrace/monkey.py
--- a/ddtrace/monkey.py
+++ b/ddtrace/monkey.py
@@ -178,7 +178,8 @@ def patch(raise_errors=True, **patch_modules):
             for m in modules_to_poi:
                 # If the module has already been imported then patch immediately
                 if m in sys.modules:
-                    patch_module(m, raise_errors=raise_errors)
+                    patch_module(module, raise_errors=raise_errors)
+                    break
                 # Otherwise, add a hook to patch when it is imported for the first time
                 else:
                     # Use factory to create handler to close over `module` and `raise_errors` values from this loop


Test patch: