Repository: DataDog/dd-trace-py

Issue:
UnicodeDecodeError with aioredis tracing enabled
### Which version of dd-trace-py are you using?

0.57.0

### Which version of pip are you using?

21.3.1

### Which version of the libraries are you using?

```
aiohttp==3.8.1
aioredis==1.3.1
aiosignal==1.2.0
amqp==5.0.9
analytics-python==1.4.0
asgiref==3.3.4
async-timeout==4.0.2
attrs==21.3.0
autobahn==21.3.1
Automat==20.2.0
aws-encryption-sdk==3.0.0
backcall==0.2.0
backoff==1.10.0
billiard==3.6.4.0
boto3==1.18.53
botocore==1.21.65
celery==5.1.2
Celery-OpenTracing==0.0.1
certifi==2021.10.8
cffi==1.15.0
channels==3.0.4
channels-redis==3.3.0
charset-normalizer==2.0.9
click==7.1.2
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
constantly==15.1.0
cryptography==36.0.1
daphne==3.0.2
ddtrace==0.53.3
decorator==5.1.0
Django==3.2.7
django-cors-headers==3.9.0
django-countries==7.2.1
django-enumfields==2.1.1
django-ipware==4.0.0
django-netfields==1.2.4
django-otp==1.1.1
django-phonenumber-field==5.2.0
django-polymorphic==3.0.0
django-ratelimit==3.0.1
django-redis==5.0.0
django-rest-auth==0.9.5
django-rest-knox==4.1.0
django-rest-polymorphic==0.1.9
django-storages==1.11.1
djangorestframework==3.12.4
drf-spectacular==0.19.0
exponent-server-sdk==2.0.0
fireblocks-sdk==1.6.4
frozenlist==1.2.0
geoip2==4.4.0
googlemaps==4.5.3
hiredis==2.0.0
hyperlink==21.0.0
idna==3.3
incremental==21.3.0
inflection==0.5.1
ipython==7.28.0
jedi==0.18.1
Jinja2==3.0.1
jmespath==0.10.0
jsonschema==4.3.2
kombu==5.2.2
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
maxminddb==2.2.0
monotonic==1.6
more-itertools==8.10.0
msgpack==1.0.3
multidict==5.2.0
netaddr==0.8.0
opentracing==2.4.0
packaging==21.3
parso==0.8.3
pdfkit==0.6.1
pexpect==4.8.0
phonenumbers==8.12.40
pickleshare==0.7.5
prompt-toolkit==3.0.24
protobuf==3.19.1
psycopg2-binary==2.9.1
ptyprocess==0.7.0
pyaes==1.6.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycoingecko==2.2.0
pycparser==2.21
pydantic==1.8.2
Pygments==2.10.0
PyJWT==1.7.1
pyOpenSSL==21.0.0
pyparsing==3.0.6
pyrsistent==0.18.0
python-dateutil==2.8.2
python-http-client==3.3.4
python-json-logger==2.0.2
pytz==2021.3
PyYAML==6.0
redis==3.5.3
requests==2.26.0
rsa==4.8
s3transfer==0.5.0
sendgrid==6.8.2
sentry-sdk==1.3.1
service-identity==21.1.0
six==1.16.0
slackclient==2.9.3
sqlparse==0.4.2
starkbank-ecdsa==2.0.3
Telethon==1.23.0
tenacity==8.0.1
traitlets==5.1.1
Twisted==21.7.0
txaio==21.2.1
typing_extensions==4.0.1
ua-parser==0.10.0
uritemplate==4.1.1
urllib3==1.26.7
user-agents==2.2.0
uWSGI==2.0.19.1
vine==5.0.0
wcwidth==0.2.5
wrapt==1.13.3
xmltodict==0.12.0
yarl==1.7.2
zope.interface==5.4.0
```

### How can we reproduce your problem?

This appears to be an issue with the args `channels-redis` sends. I don't have exact reproduction steps, but it's happening on all `group_send` calls. The issue is traced to [this line](https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/contrib/aioredis/patch.py#L110).

From Sentry, we can see that `args` is:
```
[b'EVAL', "\n                local over_capacity = 0\n                local current_time = ARGV[#ARGV - 1]\n                local expiry = ARGV[#ARGV]\n                for i=1,#KEYS do\n                    if redis.call('ZCOUNT', KEYS[i], '-inf', '+inf') < tonumber(ARGV[i + #KEYS]) then\n                        redis.call('ZADD', KEYS[i], current_time, ARGV[i])\n                        redis.call('EXPIRE', KEYS[i], expiry)\n                    else\n                        over_capacity = over_capacity + 1\n      ..., 82, 'asgispecific.01b14e3967944a38bf1a6d77860b0d9e!', 'asgispecific.1eb9eaed1ed5481992af051b3571ab0c!', 'asgispecific.eaad34e0007a4d1d94410e1ba4badf18!', 'asgispecific.f281fa62a0bc45ffaf67578bc762b519!', 'asgispecific.cf716363a9844c9ea24e19693d78eb4f!', 'asgispecific.1c41b7e2918c4a7aa9020f01639c2ceb!', 'asgispecific.60c6d5c8d88e4663bef2accc5664cfda!']
```

### What is the result that you get?

An exception is thrown:
```
UnicodeDecodeError
'utf-8' codec can't decode byte 0xb1 in position 1: invalid start byte
```

We also see,
```
UnicodeDecodeError
'utf-8' codec can't decode byte 0xee in position 1: invalid continuation byte
```

### What is the result that you expected?

No exception is thrown and the call is correctly traced.



Discussion:
Digging into this a bit more, it turns out that the issue is how `channels-redis` encodes messages. It uses [`msgpack`](https://msgpack.org/index.html) to serialize messages (python dicts in this case) when sending them via Redis.

Given that it doesn't seem like ddtrace should handle all possible encodings and Redis works with bytes, it seems that being able to decode to utf-8 is not a safe assumption.
We are experiencing this same issue and also use `channels-redis` in our project. Is there a timeline for the PR with the fix to be merged?

Resolving patch:
diff --git a/ddtrace/contrib/aioredis/patch.py b/ddtrace/contrib/aioredis/patch.py
--- a/ddtrace/contrib/aioredis/patch.py
+++ b/ddtrace/contrib/aioredis/patch.py
@@ -66,8 +66,7 @@ async def traced_execute_command(func, instance, args, kwargs):
     if not pin or not pin.enabled():
         return await func(*args, **kwargs)
 
-    decoded_args = [arg.decode() if isinstance(arg, bytes) else arg for arg in args]
-    with _trace_redis_cmd(pin, config.aioredis, instance, decoded_args):
+    with _trace_redis_cmd(pin, config.aioredis, instance, args):
         return await func(*args, **kwargs)
 
 
@@ -107,7 +106,6 @@ def traced_13_execute_command(func, instance, args, kwargs):
     if not pin or not pin.enabled():
         return func(*args, **kwargs)
 
-    decoded_args = [arg.decode() if isinstance(arg, bytes) else arg for arg in args]
     # Don't activate the span since this operation is performed as a future which concludes sometime later on in
     # execution so subsequent operations in the stack are not necessarily semantically related
     # (we don't want this span to be the parent of all other spans created before the future is resolved)
@@ -116,7 +114,7 @@ def traced_13_execute_command(func, instance, args, kwargs):
     )
 
     span.set_tag(SPAN_MEASURED_KEY)
-    query = format_command_args(decoded_args)
+    query = format_command_args(args)
     span.resource = query
     span.set_tag(redisx.RAWCMD, query)
     if pin.tags:
@@ -129,7 +127,7 @@ def traced_13_execute_command(func, instance, args, kwargs):
             redisx.DB: instance.db or 0,
         }
     )
-    span.set_metric(redisx.ARGS_LEN, len(decoded_args))
+    span.set_metric(redisx.ARGS_LEN, len(args))
     # set analytics sample rate if enabled
     span.set_tag(ANALYTICS_SAMPLE_RATE_KEY, config.aioredis.get_analytics_sample_rate())
 
@@ -157,7 +155,7 @@ async def traced_13_execute_pipeline(func, instance, args, kwargs):
 
     cmds = []
     for _, cmd, cmd_args, _ in instance._pipeline:
-        parts = [cmd.decode() if isinstance(cmd, bytes) else cmd]
+        parts = [cmd]
         parts.extend(cmd_args)
         cmds.append(format_command_args(parts))
     resource = "\n".join(cmds)
diff --git a/ddtrace/contrib/redis/util.py b/ddtrace/contrib/redis/util.py
--- a/ddtrace/contrib/redis/util.py
+++ b/ddtrace/contrib/redis/util.py
@@ -9,7 +9,10 @@
 from ...ext import SpanTypes
 from ...ext import net
 from ...ext import redis as redisx
+from ...internal.compat import binary_type
+from ...internal.compat import ensure_text
 from ...internal.compat import stringify
+from ...internal.compat import text_type
 
 
 VALUE_PLACEHOLDER = "?"
@@ -41,7 +44,10 @@ def format_command_args(args):
     out = []
     for arg in args:
         try:
-            cmd = stringify(arg)
+            if isinstance(arg, (binary_type, text_type)):
+                cmd = ensure_text(arg, errors="backslashreplace")
+            else:
+                cmd = stringify(arg)
 
             if len(cmd) > VALUE_MAX_LEN:
                 cmd = cmd[:VALUE_MAX_LEN] + VALUE_TOO_LONG_MARK
diff --git a/ddtrace/internal/compat.py b/ddtrace/internal/compat.py
--- a/ddtrace/internal/compat.py
+++ b/ddtrace/internal/compat.py
@@ -51,6 +51,7 @@
 ensure_str = six.ensure_str
 stringify = six.text_type
 string_type = six.string_types[0]
+text_type = six.text_type
 binary_type = six.binary_type
 msgpack_type = six.binary_type
 # DEV: `six` doesn't have `float` in `integer_types`


Test patch:
diff --git a/tests/contrib/aioredis/test_aioredis.py b/tests/contrib/aioredis/test_aioredis.py
--- a/tests/contrib/aioredis/test_aioredis.py
+++ b/tests/contrib/aioredis/test_aioredis.py
@@ -66,6 +66,37 @@ async def test_basic_request(redis_client):
     assert val is None
 
 
+@pytest.mark.asyncio
+@pytest.mark.snapshot
+async def test_decoding_non_utf8_args(redis_client):
+    await redis_client.set(b"\x80foo", b"\x80abc")
+    val = await redis_client.get(b"\x80foo")
+    assert val == b"\x80abc"
+
+
+@pytest.mark.asyncio
+@pytest.mark.snapshot(variants={"": aioredis_version >= (2, 0), "13": aioredis_version < (2, 0)})
+async def test_decoding_non_utf8_pipeline_args(redis_client):
+    if aioredis_version >= (2, 0):
+        p = await redis_client.pipeline(transaction=False)
+        await p.set(b"\x80blah", "boo")
+        await p.set("foo", b"\x80abc")
+        await p.get(b"\x80blah")
+        await p.get("foo")
+    else:
+        p = redis_client.pipeline()
+        p.set(b"\x80blah", "boo")
+        p.set("foo", b"\x80abc")
+        p.get(b"\x80blah")
+        p.get("foo")
+
+    response_list = await p.execute()
+    assert response_list[0] is True  # response from redis.set is OK if successfully pushed
+    assert response_list[1] is True
+    assert response_list[2].decode() == "boo"
+    assert response_list[3] == b"\x80abc"
+
+
 @pytest.mark.asyncio
 @pytest.mark.snapshot
 async def test_long_command(redis_client):
diff --git a/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_args.json b/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_args.json
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_args.json
@@ -0,0 +1,56 @@
+[[
+  {
+    "name": "redis.command",
+    "service": "redis",
+    "resource": "SET \\x80foo \\x80abc",
+    "trace_id": 0,
+    "span_id": 1,
+    "parent_id": 0,
+    "type": "redis",
+    "meta": {
+      "out.host": "127.0.0.1",
+      "redis.raw_command": "SET \\x80foo \\x80abc",
+      "runtime-id": "af49d807abff41089eed43f05d099627"
+    },
+    "metrics": {
+      "_dd.agent_psr": 1.0,
+      "_dd.measured": 1,
+      "_dd.top_level": 1,
+      "_dd.tracer_kr": 1.0,
+      "_sampling_priority_v1": 1,
+      "out.port": 6379,
+      "out.redis_db": 0,
+      "redis.args_length": 3,
+      "system.pid": 36293
+    },
+    "duration": 6351000,
+    "start": 1640819680842990000
+  }],
+[
+  {
+    "name": "redis.command",
+    "service": "redis",
+    "resource": "GET \\x80foo",
+    "trace_id": 1,
+    "span_id": 1,
+    "parent_id": 0,
+    "type": "redis",
+    "meta": {
+      "out.host": "127.0.0.1",
+      "redis.raw_command": "GET \\x80foo",
+      "runtime-id": "af49d807abff41089eed43f05d099627"
+    },
+    "metrics": {
+      "_dd.agent_psr": 1.0,
+      "_dd.measured": 1,
+      "_dd.top_level": 1,
+      "_dd.tracer_kr": 1.0,
+      "_sampling_priority_v1": 1,
+      "out.port": 6379,
+      "out.redis_db": 0,
+      "redis.args_length": 2,
+      "system.pid": 36293
+    },
+    "duration": 1959000,
+    "start": 1640819680849456000
+  }]]
diff --git a/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_pipeline_args.json b/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_pipeline_args.json
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_pipeline_args.json
@@ -0,0 +1,28 @@
+[[
+  {
+    "name": "redis.command",
+    "service": "redis",
+    "resource": "SET \\x80blah boo\nSET foo \\x80abc\nGET \\x80blah\nGET foo",
+    "trace_id": 0,
+    "span_id": 1,
+    "parent_id": 0,
+    "type": "redis",
+    "meta": {
+      "out.host": "127.0.0.1",
+      "redis.raw_command": "SET \\x80blah boo\nSET foo \\x80abc\nGET \\x80blah\nGET foo",
+      "runtime-id": "e6362b13a7394c2490fc39fa29c0e4af"
+    },
+    "metrics": {
+      "_dd.agent_psr": 1.0,
+      "_dd.measured": 1,
+      "_dd.top_level": 1,
+      "_dd.tracer_kr": 1.0,
+      "_sampling_priority_v1": 1,
+      "out.port": 6379,
+      "out.redis_db": 0,
+      "redis.pipeline_length": 4,
+      "system.pid": 60727
+    },
+    "duration": 922000,
+    "start": 1640911687480019000
+  }]]
diff --git a/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_pipeline_args_13.json b/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_pipeline_args_13.json
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.aioredis.test_aioredis.test_decoding_non_utf8_pipeline_args_13.json
@@ -0,0 +1,28 @@
+[[
+  {
+    "name": "redis.command",
+    "service": "redis",
+    "resource": "SET \\x80blah boo\nSET foo \\x80abc\nGET \\x80blah\nGET foo",
+    "trace_id": 0,
+    "span_id": 1,
+    "parent_id": 0,
+    "type": "redis",
+    "meta": {
+      "out.host": "127.0.0.1",
+      "redis.raw_command": "SET \\x80blah boo\nSET foo \\x80abc\nGET \\x80blah\nGET foo",
+      "runtime-id": "2c1fc87edd3b44ea8e46fe3854500d3c"
+    },
+    "metrics": {
+      "_dd.agent_psr": 1.0,
+      "_dd.measured": 1,
+      "_dd.top_level": 1,
+      "_dd.tracer_kr": 1.0,
+      "_sampling_priority_v1": 1,
+      "out.port": 6379,
+      "out.redis_db": 0,
+      "redis.pipeline_length": 4,
+      "system.pid": 60927
+    },
+    "duration": 982000,
+    "start": 1640911730323059000
+  }]]