Repository: elastic/apm-agent-python

Issue:
database trace show twice
maybe this is not a bug
kibana show every database trace will show twice
![image](https://user-images.githubusercontent.com/35414361/72774905-5a16a700-3c47-11ea-878c-3b05a671ec4f.png)

but in 4.2.2 version,  it show this 
![image](https://user-images.githubusercontent.com/35414361/72774972-964a0780-3c47-11ea-9d0d-19269d9259f2.png)

i think its related to span? in 4.2.2 agent won't upload span log,   please tell me how to resovle it 😢
(forgive my poor English)

**Environment (please complete the following information)**
- OS: [e.g. Linux] linux
- Python version: 3.7.
- Framework and version [e.g. Django 2.1]: Django 3.0
- APM Server version:  7.4.2
- Agent version: 5.3.3



Discussion:
It definitely seems as though the spans are being captured twice. Do you perhaps have any custom instrumentations in place? Or any manual `capture_span` calls? It's possible that in 5.3.3 we have automatic instrumentation in place that is causing duplicates with any custom/manual spans you're capturing (due to a lack of support in 4.2.2, perhaps).
we use `capture_span` to  show the `rest_framework.serializers.Serializer.to_representatio` and `celery.Task.apply_asyn` , but  they all only show once ，when i delete the `capture_span` code , the database track also twice. And  use `elasticapm.set_user_context()` to trace user authenticate  (database trace)， other is the default configuration.

capture_span code  like this 
```python
class CelerySpan(elasticapm.capture_span):
    def __call__(self, func):
        self.name = self.name or get_name_from_func(func)
        self.type = "celery.delay"

        @fn.wraps(func)
        def decorated(obj, *args, **kwds):
            self.name = f"{obj.__name__}.{func.__name__}"
            with self:
                return func(obj, *args, **kwds)
        return decorated
```
Can you please provide a `pip list` for us so we can see what you have installed? This will help us narrow down the source of the extra spans.

I assume that only `elastic-apm` was upgraded? The rest of the dependencies are the same?
pipenv graph
```
[packages]
attrs = "==19.3.0"
autobahn = "==20.1.2"
celery = "==4.4.0"
cffi = "==1.13.2"
chinesecalendar = "==1.3.1"
cryptography = "==2.8"
django = "==3.0.2"
django-cors-headers = "==3.2.1"
django-mysql = "==3.3.0"
django-mysql-geventpool = "==0.2.5"
django-s3-storage = "==0.13.0"
djangorestframework = "==3.11.0"
drf-nested-routers = "==0.91"
drf-yasg = "==1.17.0"
elastic-apm = "==5.3.3"
emoji = "==0.5.4"
gevent = "==1.4.0"
hutils = "==0.6.1"
inflection = "==0.3.1"
jsonpickle = "==1.2"
kombu = "==4.6.7"
lupa = "==1.9"
mongoengine = "==0.19.1"
pillow = "==7.0.0"
pyjwt = "==1.7.1"
pymysql = "==0.9.3"
python-dateutil = "==2.8.0"
pytz = "==2019.3"
pywin32 = {version = "==227",sys_platform = "== 'win32'"}
qiniu = "==7.2.6"
qrcode = "==6.1"
redis = "==3.3.11"
rx = "==3.0.1"
sentry-sdk = "==0.14.0"
sqlalchemy = "==1.3.12"
tenacity = "==6.0.0"
wechatpy = "==1.8.6"
xlrd = "==1.2.0"
xlwt = "==1.3.0"
xmltodict = "==0.12.0"

[dev-packages]
black = "==19.10b0"
fabric3 = "==1.14.post1"
factory-boy = ">=2.12.0"
faker = ">=3.0.0"
fakeredis = ">=1.0"
flake8 = ">=3.7.0"
flake8-import-order = ">=0.18"
freezegun = ">=0.3.13"
ipython = ">=7.11.1"
isort = {version = "*",extras = ["pipfile"]}
mongomock = ">=3.17.0"
pook = ">=0.2.8"
pycodestyle = ">=2.4.0"
pyflakes = ">=2.0.0"
pylint = ">=2.2.2"
pylint-django = ">=2.0.2"
pylint-topology = ">=0.1.1"
pytest-cov = ">=2.8.1"
pytest-django = ">=3.8.0"
pytest-xdist = ">=1.31.0"
```
Nothing super obvious there.

I had another thought. Can you show me the stack trace frames for two of the duplicate spans? Like this:

![span](https://user-images.githubusercontent.com/702318/73008148-2c5a6b00-3dcb-11ea-982b-c49c97c6f8be.gif)

an example, the duplicate spans all like this 

![image](https://user-images.githubusercontent.com/35414361/73039129-e5827900-3e8f-11ea-90af-1b4801be8e0f.png)

![image](https://user-images.githubusercontent.com/35414361/73039170-077bfb80-3e90-11ea-8018-2bb4b94e3915.png)

Unfortunately your screenshots only show one of the spans -- what I'd like to do is compare the frames of two duplicate spans, see if I can spot why it's being instrumented twice. Can you add the duplicate to the span shown above?
a new api from local server
![screencapture-span1](https://user-images.githubusercontent.com/35414361/73120660-004a1000-3fac-11ea-8e9e-277b66bd252d.png)

this is the deplicate spans
![screencapture-span2](https://user-images.githubusercontent.com/35414361/73120661-00e2a680-3fac-11ea-9aa1-27d944f5d24a.png)

Hi @zzzzzzmj 

As @basepi says, this kind of looks like we instrument things twice. That shouldn't happen, as we check before instrumenting. But there could be bugs with those checks of course.

If possible, could you add this code in a view somewhere and see if the assertion fails? Of course only on a test system :)

    from urllib.request import AbstractHTTPHandler
    assert not hasattr(AbstractHTTPHandler.do_open._self_wrapper, "_self_wrapper")


I add the code in a view and the request is also 200. @beniwohli 

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/base.py b/elasticapm/instrumentation/packages/base.py
--- a/elasticapm/instrumentation/packages/base.py
+++ b/elasticapm/instrumentation/packages/base.py
@@ -38,6 +38,11 @@
 logger = get_logger("elasticapm.instrument")
 
 
+class ElasticAPMFunctionWrapper(wrapt.FunctionWrapper):
+    # used to differentiate between our own function wrappers and 1st/3rd party wrappers
+    pass
+
+
 class AbstractInstrumentedModule(object):
     """
     This class is designed to reduce the amount of code required to
@@ -136,8 +141,13 @@ def instrument(self):
                     # We jump through hoop here to get the original
                     # `module`/`method` in the call to `call_if_sampling`
                     parent, attribute, original = wrapt.resolve_path(module, method)
+                    if isinstance(original, ElasticAPMFunctionWrapper):
+                        logger.debug("%s.%s already instrumented, skipping", module, method)
+                        continue
                     self.originals[(module, method)] = original
-                    wrapper = wrapt.FunctionWrapper(original, functools.partial(self.call_if_sampling, module, method))
+                    wrapper = ElasticAPMFunctionWrapper(
+                        original, functools.partial(self.call_if_sampling, module, method)
+                    )
                     wrapt.apply_patch(parent, attribute, wrapper)
                     instrumented_methods.append((module, method))
                 except ImportError:


Test patch:
diff --git a/tests/instrumentation/base_tests.py b/tests/instrumentation/base_tests.py
--- a/tests/instrumentation/base_tests.py
+++ b/tests/instrumentation/base_tests.py
@@ -37,6 +37,7 @@
 import pytest
 
 import elasticapm
+from elasticapm.conf import constants
 from elasticapm.conf.constants import SPAN
 from elasticapm.instrumentation.packages.base import AbstractInstrumentedModule
 from elasticapm.utils import compat, wrapt
@@ -65,7 +66,8 @@ class _TestDummyInstrumentation(AbstractInstrumentedModule):
     def call(self, module, method, wrapped, instance, args, kwargs):
         kwargs = kwargs or {}
         kwargs["call_args"] = (module, method)
-        return wrapped(*args, **kwargs)
+        with elasticapm.capture_span("dummy"):
+            return wrapped(*args, **kwargs)
 
 
 def test_instrument_nonexisting_method_on_module():
@@ -79,6 +81,25 @@ def test_instrument_nonexisting_method(caplog):
     assert "has no attribute" in record.message
 
 
+def test_double_instrument(elasticapm_client):
+    elasticapm_client.begin_transaction("test")
+    inst = _TestDummyInstrumentation()
+    try:
+        inst.instrument()
+        assert hasattr(Dummy.dummy, "_self_wrapper")
+        Dummy().dummy()
+        elasticapm_client.end_transaction()
+        assert len(elasticapm_client.spans_for_transaction(elasticapm_client.events[constants.TRANSACTION][0])) == 1
+        inst.instrumented = False
+        inst.instrument()
+        elasticapm_client.begin_transaction("test")
+        Dummy().dummy()
+        elasticapm_client.end_transaction()
+        assert len(elasticapm_client.spans_for_transaction(elasticapm_client.events[constants.TRANSACTION][1])) == 1
+    finally:
+        inst.uninstrument()
+
+
 @pytest.mark.skipif(compat.PY3, reason="different object model")
 def test_uninstrument_py2(caplog):
     assert isinstance(Dummy.dummy, types.MethodType)