Repository: DataDog/dd-trace-py

Issue:
Proposal: Avoid nested db spans in django integration 
# Background

The Django integration includes additional instrumentation for Django database backends. The Django database instrumentation results in two spans for any query executions if the database library has also been patched. For instance, if a mysql integration has been enabled in the tracer, a user will see the following in their traces:

![image](https://user-images.githubusercontent.com/155673/118178366-dff0cf80-b401-11eb-851f-e5501f0febe7.png)

The implementation of this is handled as follows:
1. the `django.db.connections` collection is [patched](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/django/patch.py#L101) such that any new connection is [patched](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/django/patch.py#L95) so that the cursor function will [wrapped](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/django/patch.py#L85) with the `TracedCursor` provided by the `dbapi` integration.
2. the `TracedCursor` wraps a cursor object such that a [query execution will within a span](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/dbapi/__init__.py#L76).
3. if the db cursor wrapped with `TracedCursor` by the Django integration is already wrapped, then there will be two spans created, the parent span created by the `TracedCursor` from the Django integration and the child span created by the wrapped cursor from any patched database library.

# Motivation

## Unnecessary

* The Django span timing is not meaningfully different.
* The Django span metadata is redundant.

## Bugs

* Coercion: while the psycopg integration [coerces](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/psycopg/patch.py#L60) a composed query, the db span that is created by Django does not perform the same coercion when it's setting the span resource. The `TracedCursor` from the dbapi integration [sets the resource](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/dbapi/__init__.py#L62) to the `query` without any coercion. If this coercion does not occur, the encoding of the trace using our custom msgpack encoder will cause an error to be raised. While the psycopg integration has been fixed to address this bug with the custom msgpack encoder, the same was not applied to the dbapi integration nor the Django db backend instrumentation.

# Proposal

1. Change the Django db instrumentation so as to avoid wrapping a cursor object that has already been wrapped.
2. This will mean that Django users by default will not see two spans for db calls.
TypeError: can not serialize 'Composed' object
### Which version of dd-trace-py are you using?

0.46, having upgraded from 0.43

### Which version of the libraries are you using?

`protobuf==3.14.0`
`tenacity==6.3.1`

and relevantly:

`psycopg2-binary==2.8.6`

### How can we reproduce your problem?

Send a trace with a psycopg2

### What is the result that you get?

On Sentry after deploying the ddtrace upgrade, immediately got:

`TypeError: can not serialize 'Composed' object`

This looks like a recurrence of #732.

Stack frames:

ddtrace/internal/_encoding.pyx in ddtrace.internal._encoding.Packer._pack at line 263
ddtrace/internal/_encoding.pyx in ddtrace.internal._encoding.Packer._pack at line 187
ddtrace/internal/_encoding.pyx in ddtrace.internal._encoding.Packer.pack at line 269
ddtrace/internal/_encoding.pyx in ddtrace.internal._encoding.Packer.pack at line 272
ddtrace/internal/_encoding.pyx in ddtrace.internal._encoding.MsgpackEncoder.encode_trace at line 300
ddtrace/internal/_encoding.pyx in ddtrace.internal._encoding.MsgpackEncoder.encode_trace at line 299
ddtrace/internal/writer.py in write at line 257


### What is the result that you expected?

No error.



Discussion:
> Change the Django db instrumentation so as to avoid wrapping a cursor object that has already been wrapped.

Are we able to reliably detect this?

I assume we prefer to keep the Psycopg2 spans instead of the Django db one?

Does patching order matter? When do we patch the Django connection vs Psycopg2?

(btw, this proposal makes sense to me)
Yeah same concern as Brett. IIRC Django does its own wrapping of cursors which makes it hard to access the original to check if it is patched by us.

> The Django span timing is not meaningfully different.

Are we certain of this? If Django is wrapping cursors itself then it might be doing some additional logic which could be meaningful. Might be good to verify.

@brettlangdon we should be able to do this reliably, we just need to be unwrap the cursor as Django [wraps](https://github.com/django/django/blob/9d67bfadf897d4eb082b398fe9482fc6753c7bf2/django/db/backends/base/base.py#L552) cursors.

We'd want the db spans from psycopg2 rather than the ones from Django.

The patching would occur in both cases but the Django patched db code would skip creating a span if the db cursor has already been patched by us.



> > The Django span timing is not meaningfully different.
> 
> Are we certain of this? If Django is wrapping cursors itself then it might be doing some additional logic which could be meaningful. Might be good to verify.

@Kyle-Verhoog The Django DB span covers the execution of a query with a cursor. Any difference in timings would occur in the Django's [CursorWrapper](https://github.com/django/django/blob/9d67bfadf897d4eb082b398fe9482fc6753c7bf2/django/db/backends/utils.py#L16-L69) which does some error handling. Most db backends use this wrapper as is. Sqlite does have its [own wrapper](https://github.com/django/django/blob/9d67bfadf897d4eb082b398fe9482fc6753c7bf2/django/db/backends/sqlite3/base.py#L333-L337) which does some query conversion using some regex. Any timing difference would be accounted for by the Django library calls rather than some user code, which seems like less meaningful when looking at a trace for a slow web request. Though I might be wrong about that and we should consider how to accurately represent the work the Django db backends are doing in addition to the underlying db libraries. 
It seems reasonable to me to provide this solution as a default while still being able to opt into Django database spans in case extra work is done in the wrapped cursors. :+1: 
Agreed, this sounds like a reasonable proposal.
This proposal would be a breaking change.
Are you by any chance using the deprecated connection factor rather than patching as described in https://ddtrace.readthedocs.io/en/stable/integrations.html#module-ddtrace.contrib.psycopg? The deprecated approach is not converting the query object to a string:

https://github.com/DataDog/dd-trace-py/blob/97140a6f72b241982454feac64816f57b6ba6bc1/ddtrace/contrib/psycopg/connection.py#L53
No we use plain `ddtrace-run` and let it call `patch_all()`.
I am adding an encoding test to psycopg in #2003 which is passing. Mind taking a look over the test case? Maybe we are still missing something.

https://github.com/DataDog/dd-trace-py/pull/2003/files#diff-41f07c055df91a25d2f3c15e89367264e67c74167119355a7b1f8f9da3bf9546R297-R313
Hmm. Looking a little closer at the crash it happened during our Django migrations phase, and *may* have been triggered by this:

```
cursor.execute(
            "SET statement_timeout=%s",
            (timeout_milliseconds,),
        )
```

Or by this:

```
cursor.execute(
    """
    CREATE OR REPLACE FUNCTION trim_postcode (postcode text) RETURNS text AS $$
    declare
      trimmed_postcode text;
    BEGIN
      trimmed_postcode := REPLACE(postcode, ' ', '');
      IF LENGTH(trimmed_postcode) > 5 THEN
        RETURN SUBSTRING(trimmed_postcode, 1, LENGTH(trimmed_postcode) - 3);
      ELSE
        RETURN trimmed_postcode;
      END IF;
    END; $$
    LANGUAGE PLPGSQL;
    """
)
```

Hard to know because Sentry couldn't capture the line within our code, only within the datadog thread. But it did capture queries in its "breadcrumbs" feature, up to the first one there.

Of course those might be completely off if the breadcrumbs are wrong, and I think they sometimes are.  We do directly use the `pscyopg2.sql` classes later in the migration process, with some formatting calls like:

```
from psycopg2 import sql

sql.SQL("SELECT {x}").format(x=sql.Literal("x"))
```

Hopefully one of these can reproduce the crash.
@adamchainz Could you try the branch in #2229 to see if this resolves the problem you are having? I have not been able to reproduce this issue in a test though wanted to share on potential mitigation before proceeding down this path.
I haven't had time to check this out. It's a little hard for us since it only happens in the production environment. Did you try testing the example code as above?
@adamchainz I have been looking at this recently and think I might know why this is occurring for you and others.

You are using Django here and I assume you have not explicitly disabled the Django DB instrumentation that our integration provides. Does this problem go away for you if you disable that instrumentation with `ddtrace.config.django['instrument_databases'] = False`? This is described in [the Django integration configuration](https://ddtrace.readthedocs.io/en/stable/integrations.html#id21).

The Django DB instrumentation creates a span in addition to the span created by any dbapi-compliant library that has been patched as well. While the psycopg integration [coerces](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/psycopg/patch.py#L60) a composed query, the db span that is created by Django will not perform the same coercion when it's setting the span resource. This is because Django [wrapping](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/django/patch.py#L85) creates a cursor for its db connection using the `TracedCursor` from the `dbapi` integration, [setting the resource](https://github.com/DataDog/dd-trace-py/blob/d5b84311858870411c4a4c13486dc32b8e29477f/ddtrace/contrib/dbapi/__init__.py#L62) to the `query` without any coercion. 

If this is the source of your issue, we could resolve this by performing such coercions in the Django DB instrumentation (which might be tricky since we'd have to handle multiple backends) or we just avoid creating an additional span for the cursor call if that cursor has already been wrapped because of a db integration. I'm inclined to pursue the latter option if this is in fact the source of this issue.
I have opened an issue for discussion of a proposal to resolve the issue with Django db instrumentation: #2438 
> Does this problem go away for you if you disable that instrumentation with `ddtrace.config.django['instrument_databases'] = False`?

I just got around to testing this, and yes, the problem does go away with that change.

I tested with this one-liner:

`ddtrace-run python manage.py shell -c 'from django.db import connection; from psycopg2 import sql; cursor = connection.cursor(); cursor.execute(sql.SQL("SELECT 123"))'`


Resolving patch:
diff --git a/ddtrace/contrib/django/patch.py b/ddtrace/contrib/django/patch.py
--- a/ddtrace/contrib/django/patch.py
+++ b/ddtrace/contrib/django/patch.py
@@ -19,6 +19,16 @@
 from ddtrace.constants import SPAN_MEASURED_KEY
 from ddtrace.contrib import dbapi
 from ddtrace.contrib import func_name
+
+
+try:
+    from psycopg2._psycopg import cursor as psycopg_cursor_cls
+
+    from ddtrace.contrib.psycopg.patch import Psycopg2TracedCursor
+except ImportError:
+    psycopg_cursor_cls = None
+    Psycopg2TracedCursor = None
+
 from ddtrace.ext import SpanTypes
 from ddtrace.ext import http
 from ddtrace.ext import sql as sqlx
@@ -82,7 +92,15 @@ def cursor(django, pin, func, instance, args, kwargs):
             "django.db.alias": alias,
         }
         pin = Pin(service, tags=tags, tracer=pin.tracer, app=prefix)
-        return dbapi.TracedCursor(func(*args, **kwargs), pin, config.django)
+        cursor = func(*args, **kwargs)
+        traced_cursor_cls = dbapi.TracedCursor
+        if (
+            Psycopg2TracedCursor is not None
+            and hasattr(cursor, "cursor")
+            and isinstance(cursor.cursor, psycopg_cursor_cls)
+        ):
+            traced_cursor_cls = Psycopg2TracedCursor
+        return traced_cursor_cls(cursor, pin, config.django)
 
     if not isinstance(conn.cursor, wrapt.ObjectProxy):
         conn.cursor = wrapt.FunctionWrapper(conn.cursor, trace_utils.with_traced_module(cursor)(django))


Test patch:
diff --git a/tests/contrib/django/test_django_snapshots.py b/tests/contrib/django/test_django_snapshots.py
--- a/tests/contrib/django/test_django_snapshots.py
+++ b/tests/contrib/django/test_django_snapshots.py
@@ -68,3 +68,38 @@ def test_safe_string_encoding(client):
 )
 def test_404_exceptions(client):
     assert client.get("/404-view/").status_code == 404
+
+
+@pytest.fixture()
+def psycopg2_patched(transactional_db):
+    from django.db import connections
+
+    from ddtrace.contrib.psycopg.patch import patch
+    from ddtrace.contrib.psycopg.patch import unpatch
+
+    patch()
+
+    # # force recreate connection to ensure psycopg2 patching has occurred
+    del connections["postgres"]
+    connections["postgres"].close()
+    connections["postgres"].connect()
+
+    yield
+
+    unpatch()
+
+
+@snapshot(ignores=["meta.out.host"])
+@pytest.mark.django_db
+def test_psycopg_query_default(client, psycopg2_patched):
+    """Execute a psycopg2 query on a Django database wrapper"""
+    from django.db import connections
+    from psycopg2.sql import SQL
+
+    query = SQL("""select 'one' as x""")
+    conn = connections["postgres"]
+    with conn.cursor() as cur:
+        cur.execute(query)
+        rows = cur.fetchall()
+        assert len(rows) == 1, rows
+        assert rows[0][0] == "one"
diff --git a/tests/snapshots/tests.contrib.django.test_django_snapshots.test_psycopg_query_default.snap b/tests/snapshots/tests.contrib.django.test_django_snapshots.test_psycopg_query_default.snap
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.django.test_django_snapshots.test_psycopg_query_default.snap
@@ -0,0 +1,38 @@
+[[{"name" "postgres.query"
+   "service" "postgresdb"
+   "resource" "select 'one' as x"
+   "type" "sql"
+   "error" 0
+   "span_id" 0
+   "trace_id" 0
+   "parent_id" nil
+   "start" 1623177658188007300
+   "duration" 1828500
+   "meta" {"runtime-id" "98e7426d5d6d44d1a5d76d1322d69cac"
+           "django.db.vendor" "postgresql"
+           "django.db.alias" "postgres"}
+   "metrics" {"_dd.agent_psr" 1.0
+              "_dd.measured" 1
+              "_sampling_priority_v1" 1
+              "system.pid" 236
+              "db.rowcount" 1
+              "sql.rows" 1
+              "_dd.tracer_kr" 1.0}}
+      {"name" "postgres.query"
+       "service" "postgres"
+       "resource" "select 'one' as x"
+       "type" "sql"
+       "error" 0
+       "span_id" 1
+       "trace_id" 0
+       "parent_id" 0
+       "start" 1623177658188405100
+       "duration" 1265000
+       "meta" {"out.host" "127.0.0.1"
+               "db.name" "test_postgres"
+               "db.user" "postgres"
+               "db.application" "None"}
+       "metrics" {"_dd.measured" 1
+                  "out.port" 5432
+                  "db.rowcount" 1
+                  "sql.rows" 1}}]]