Repository: DataDog/dd-trace-py

Issue:
failed to send traces to Datadog Agent
### Summary of problem
Datadog Agent version: 7.42.0
Python 3.11
`failed to send traces to Datadog Agent` errors appears when updating ddtrace to version 1.7.3
If I set the environment variable DD_TRACE_API_VERSION=v0.4, then the errors disappear and everything works

Also, I found a strange thing in the payload when I enabled the _DD_TRACE_WRITER_LOG_ERROR_PAYLOADS:
payload starts with`\x92\xdc\x00 \xa0\xa3srv` instead of `\x92\xdc\x00\x1f\xa0\xa3srv`

### Which version of dd-trace-py are you using?
ddtrace==1.7.3

### Which version of pip are you using?
pip==22.3.1

### Which libraries and their versions are you using?
<details>
  <summary>`pip freeze`</summary>
**aiodns==3.0.0 
aiohttp==3.8.3
aiosignal==1.3.1
async-timeout==4.0.2
attrs==22.2.0
Brotli==1.0.9
bytecode==0.14.0
cattrs==22.2.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
datadog==0.44.0
ddsketch==2.0.4
ddtrace==1.7.3
envier==0.4.0
frozenlist==1.3.3
idna==3.4
jsonschema==4.17.3
multidict==6.0.4
packaging==23.0
protobuf==4.21.12
pycares==4.3.0
pycparser==2.21
pyrsistent==0.19.3
requests==2.28.2
six==1.16.0
tenacity==8.1.0
typing_extensions==4.4.0
urllib3==1.26.14
xmltodict==0.13.0
yarl==1.8.2**
</details>

### How can we reproduce your problem?
send traces using api version == v0.5

### What is the result that you get?
ddtrace error
`
failed to send traces to Datadog Agent at http://10.188.217.158:8126/v0.5/traces: HTTP error status 400, reason Bad Request, 16 additional messages skipped
`
datadog agent error
`Cannot decode v0.5 traces payload: msgp: attempted to decode type "array" with method for "str"
`
### What is the result that you expected?
Same behavior as DD_TRACE_API_VERSION=v0.4


Discussion:
Hi @grmnz , do you have a reproduction that you could share? I tried recreating this with the same Datadog agent, Python, and tracer version but was unable to. 
Looking into the payload info you provided, it looks like the origin key is missing as the second entry in the string table. Could you possibly send me the full payload? zach.groves@datadoghq.com
@grmnz Is `PYTHONOPTIMIZE` set in your environment, or are you using the `-O` option, by any chance?
thanks for the help @ZStriker19 @P403n1x87 ! 
Yes, I use the -OO flag
It looks like without this option everything is fine (DD_TRACE_API_VERSION=v0.5)

@ZStriker19 
without -OO option I see `_dd.origin key` in the payload (`b'\x92\xdc\x00 \xa0\xaa_dd.origin\xa3srv`)
with --OO option payload is `b'\x92\xdc\x00\x1f\xa0\xa3srv`
but locally I can not reproduce the problem with the length of the array, in both cases, the length of the array in the payload is correctly
of the above, do I still need to send the full payload?
@grmnz thanks for the new details. I think this should be enough as we have located the potential source of the problem. We will now prepare a fix for it.

Resolving patch:
diff --git a/riotfile.py b/riotfile.py
--- a/riotfile.py
+++ b/riotfile.py
@@ -279,20 +279,23 @@ def select_pys(min_version=MIN_PYTHON_VERSION, max_version=MAX_PYTHON_VERSION):
         ),
         Venv(
             name="tracer",
+            pkgs={
+                "msgpack": latest,
+                "attrs": ["==20.1.0", latest],
+                "structlog": latest,
+                # httpretty v1.0 drops python 2.7 support
+                "httpretty": "==0.9.7",
+            },
+            # Riot venvs break with Py 3.11 importlib, specifically with hypothesis (test_http.py).
+            # We'll skip the test_http.py tests in riot and run them separately through tox in CI.
+            # See linked riot issue: https://github.com/DataDog/riot/issues/192
+            command="pytest {cmdargs} tests/tracer/ --ignore=tests/tracer/test_http.py",
             venvs=[
+                Venv(pys=select_pys()),
                 Venv(
-                    pys=select_pys(),
-                    pkgs={
-                        "msgpack": latest,
-                        "attrs": ["==20.1.0", latest],
-                        "structlog": latest,
-                        # httpretty v1.0 drops python 2.7 support
-                        "httpretty": "==0.9.7",
-                    },
-                    # Riot venvs break with Py 3.11 importlib, specifically with hypothesis (test_http.py).
-                    # We'll skip the test_http.py tests in riot and run them separately through tox in CI.
-                    # See linked riot issue: https://github.com/DataDog/riot/issues/192
-                    command="pytest {cmdargs} tests/tracer/ --ignore=tests/tracer/test_http.py",
+                    env={"PYTHONOPTIMIZE": "1"},
+                    # Test with the latest version of Python only
+                    pys=".".join((str(_) for _ in SUPPORTED_PYTHON_VERSIONS[-1])),
                 ),
             ],
         ),


Test patch:
diff --git a/tests/tracer/test_encoders.py b/tests/tracer/test_encoders.py
--- a/tests/tracer/test_encoders.py
+++ b/tests/tracer/test_encoders.py
@@ -547,23 +547,28 @@ def filter_mut(ts):
     ]
 
 
-def string_table_test(t, offset=0):
-    assert len(t) == 1 + offset
+def string_table_test(t, origin_key=False):
+    assert len(t) == 1 + origin_key
+
+    assert 0 == t.index("")
+    if origin_key:
+        assert 1 == t.index(ORIGIN_KEY)
+
     id1 = t.index("foobar")
-    assert len(t) == 2 + offset
+    assert len(t) == 2 + origin_key
     assert id1 == t.index("foobar")
-    assert len(t) == 2 + offset
+    assert len(t) == 2 + origin_key
     id2 = t.index("foobaz")
-    assert len(t) == 3 + offset
+    assert len(t) == 3 + origin_key
     assert id2 == t.index("foobaz")
-    assert len(t) == 3 + offset
+    assert len(t) == 3 + origin_key
     assert id1 != id2
 
 
 def test_msgpack_string_table():
     t = MsgpackStringTable(1 << 10)
 
-    string_table_test(t, offset=1)
+    string_table_test(t, origin_key=1)
 
     size = t.size
     encoded = t.flush()