Repository: getsentry/sentry-python

Issue:
Bytes Objects serialized as list of integers in Python 3
Using `sentry_sdk` version 0.13.0 in Python 3.7.3, strings encoded as `bytes` objects are serialized as lists of integers:

```
>>> from sentry_sdk.serializer import serialize
>>> serialize(b'123abc')
[49, 50, 51, 97, 98, 99]
>>> serialize(b'123abc', should_repr_strings=True)
['49', '50', '51', '97', '98', '99']
```

This doesn't seem like the expected behavior, instead it could be:
```
>>> serialize(b'123abc')
'123abc'
>>> serialize(b'123abc', should_repr_strings=True)
"b'123abc'"
```

I was able to reproduce the issue with Python 3.7.3, 3.6.7, 3.5.6 and 3.4.10. The serialization seems to behave properly with Python 2.7.


Discussion:
I would like this to be `"b'...'"` on Python 3 and `"'...'"` on Python 2.
It looks like that's already the case:
```
Python 2.7.15 (default, Mar 21 2019, 11:27:54) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sentry_sdk.serializer import serialize
>>> serialize('123abc')
u'123abc'
>>> serialize('123abc', should_repr_strings=True)
u"'123abc'"
```
Right, the bugfix would affect py3 only. Do you want to make a PR? Otherwise I'll do it next week
I'm currently working on it, the fix for the serializer itself should be pretty simple, it's just about making sure we're testing for `bytes`.

The bigger challenge is to fix `safe_repr` so that it escapes non-printable characters. Thankfully `repr` in Python 3 has the good behavior, but fixing this in Python 2 without breaking support for non-latin characters is proving to be challenging.
I'm still working on `safe_repr` at the moment. The problem with this tool is that does not safely serialize control characters such as `\n` or `\x00`.

_Edit:_ This is relevant because fixing the `bytes` serialization will make use of `safe_repr` instead, possibly handling random binary data.

With Python 2.7.15:
```py
>>> from sentry_sdk.utils import safe_repr
>>> safe_repr(u'лошадь\n\x00')
u"u'\u043b\u043e\u0448\u0430\u0434\u044c\n\x00'"
>>> safe_repr(u'лошадь\n\x00'.encode('utf-8'))
u"'\xd0\xbb\xd0\xbe\xd1\x88\xd0\xb0\xd0\xb4\xd1\x8c\n\x00'"
```

With Python 3.7.3:
```py
>>> from sentry_sdk.utils import safe_repr
>>> safe_repr(u'лошадь\n\x00')
"'лошадь\\n\\x00'"  # This is correct
>>> safe_repr(u'лошадь\n\x00'.encode('utf-8'))
"b'Ð»Ð¾Ñ\x88Ð°Ð´Ñ\x8c\n\x00'"
```

Fixing this for Python 3 is very straightforward, all we need is `repr`. For Python 2 however that would break the current behavior where non-ascii characters are preserved in the output:
https://github.com/getsentry/sentry-python/blob/b784d63d059cecd3b2dc20a6f865d7a47b6ebdad/tests/utils/test_general.py#L35-L36

I can think of three ways to go at this issue:
1. Only fix the behavior in Py3, leave the Py2 behavior (and bugs) as-is
2. Write a custom new `repr` function for Py2 that back-ports the Py3 behavior (outputs unicode, and only escape non-printable characters). This might come at the cost of performance, unless there's a binary out there that we are willing to add to the dependencies.
3. Just use the `repr` output as-is in all cases, which will be a regression for Python 2.

What's the best option, given the functional priorities of the SDK? Is this worth fixing as a separate issue maybe?
I think it's best to only fix those bugs in Py3.
Good, I'll do that, thanks for the guidance!

Resolving patch:
diff --git a/sentry_sdk/serializer.py b/sentry_sdk/serializer.py
--- a/sentry_sdk/serializer.py
+++ b/sentry_sdk/serializer.py
@@ -36,11 +36,17 @@
     # Importing ABCs from collections is deprecated, and will stop working in 3.8
     # https://github.com/python/cpython/blob/master/Lib/collections/__init__.py#L49
     from collections import Mapping, Sequence
+
+    serializable_str_types = string_types
+
 else:
     # New in 3.3
     # https://docs.python.org/3/library/collections.abc.html
     from collections.abc import Mapping, Sequence
 
+    # Bytes are technically not strings in Python 3, but we can serialize them
+    serializable_str_types = (str, bytes)
+
 MAX_DATABAG_DEPTH = 5
 MAX_DATABAG_BREADTH = 10
 CYCLE_MARKER = u"<cyclic>"
@@ -285,7 +291,7 @@ def _serialize_node_impl(
 
             return rv_dict
 
-        elif not isinstance(obj, string_types) and isinstance(obj, Sequence):
+        elif not isinstance(obj, serializable_str_types) and isinstance(obj, Sequence):
             rv_list = []
 
             for i, v in enumerate(obj):
diff --git a/sentry_sdk/utils.py b/sentry_sdk/utils.py
--- a/sentry_sdk/utils.py
+++ b/sentry_sdk/utils.py
@@ -348,32 +348,42 @@ def safe_str(value):
         return safe_repr(value)
 
 
-def safe_repr(value):
-    # type: (Any) -> str
-    try:
-        rv = repr(value)
-        if isinstance(rv, bytes):
-            rv = rv.decode("utf-8", "replace")
-
-        # At this point `rv` contains a bunch of literal escape codes, like
-        # this (exaggerated example):
-        #
-        # u"\\x2f"
-        #
-        # But we want to show this string as:
-        #
-        # u"/"
+if PY2:
+
+    def safe_repr(value):
+        # type: (Any) -> str
         try:
-            # unicode-escape does this job, but can only decode latin1. So we
-            # attempt to encode in latin1.
-            return rv.encode("latin1").decode("unicode-escape")
+            rv = repr(value).decode("utf-8", "replace")
+
+            # At this point `rv` contains a bunch of literal escape codes, like
+            # this (exaggerated example):
+            #
+            # u"\\x2f"
+            #
+            # But we want to show this string as:
+            #
+            # u"/"
+            try:
+                # unicode-escape does this job, but can only decode latin1. So we
+                # attempt to encode in latin1.
+                return rv.encode("latin1").decode("unicode-escape")
+            except Exception:
+                # Since usually strings aren't latin1 this can break. In those
+                # cases we just give up.
+                return rv
         except Exception:
-            # Since usually strings aren't latin1 this can break. In those
-            # cases we just give up.
-            return rv
-    except Exception:
-        # If e.g. the call to `repr` already fails
-        return u"<broken repr>"
+            # If e.g. the call to `repr` already fails
+            return u"<broken repr>"
+
+
+else:
+
+    def safe_repr(value):
+        # type: (Any) -> str
+        try:
+            return repr(value)
+        except Exception:
+            return "<broken repr>"
 
 
 def filename_for_module(module, abs_path):


Test patch:
diff --git a/tests/test_serializer.py b/tests/test_serializer.py
--- a/tests/test_serializer.py
+++ b/tests/test_serializer.py
@@ -1,5 +1,5 @@
 from datetime import datetime
-
+import sys
 
 import pytest
 
@@ -30,3 +30,39 @@ def test_datetime_precision(dt, semaphore_normalize):
         # Float glitches can happen, and more glitches can happen
         # because we try to work around some float glitches in semaphore
         assert (dt - dt2).total_seconds() < 1.0
+
+    @given(binary=st.binary(min_size=1))
+    def test_bytes_serialization_decode_many(binary, message_normalizer):
+        result = message_normalizer(binary, should_repr_strings=False)
+        assert result == binary.decode("utf-8", "replace")
+
+    @given(binary=st.binary(min_size=1))
+    def test_bytes_serialization_repr_many(binary, message_normalizer):
+        result = message_normalizer(binary, should_repr_strings=True)
+        assert result == repr(binary)
+
+
+@pytest.fixture
+def message_normalizer(semaphore_normalize):
+    if semaphore_normalize({"test": "test"}) is None:
+        pytest.skip("no semaphore available")
+
+    def inner(message, **kwargs):
+        event = serialize({"logentry": {"message": message}}, **kwargs)
+        normalized = semaphore_normalize(event)
+        return normalized["logentry"]["message"]
+
+    return inner
+
+
+def test_bytes_serialization_decode(message_normalizer):
+    binary = b"abc123\x80\xf0\x9f\x8d\x95"
+    result = message_normalizer(binary, should_repr_strings=False)
+    assert result == u"abc123\ufffd\U0001f355"
+
+
+@pytest.mark.xfail(sys.version_info < (3,), reason="Known safe_repr bugs in Py2.7")
+def test_bytes_serialization_repr(message_normalizer):
+    binary = b"abc123\x80\xf0\x9f\x8d\x95"
+    result = message_normalizer(binary, should_repr_strings=True)
+    assert result == r"b'abc123\x80\xf0\x9f\x8d\x95'"
diff --git a/tests/utils/test_general.py b/tests/utils/test_general.py
--- a/tests/utils/test_general.py
+++ b/tests/utils/test_general.py
@@ -36,6 +36,19 @@ def test_safe_repr_regressions():
     assert u"лошадь" in safe_repr(u"лошадь")
 
 
+@pytest.mark.xfail(
+    sys.version_info < (3,),
+    reason="Fixing this in Python 2 would break other behaviors",
+)
+@pytest.mark.parametrize("prefix", (u"", u"abcd", u"лошадь"))
+@pytest.mark.parametrize("character", u"\x00\x07\x1b\n")
+def test_safe_repr_non_printable(prefix, character):
+    """Check that non-printable characters are escaped"""
+    string = prefix + character
+    assert character not in safe_repr(string)
+    assert character not in safe_repr(string.encode("utf-8"))
+
+
 def test_abs_path():
     """Check if abs_path is actually an absolute path. This can happen either
     with eval/exec like here, or when the file in the frame is relative to