Repository: DataDog/dd-trace-py

Issue:
Either opentracing SpanContextCorruptedException or no child spans
Using the ddtracer.opentracing exactly as in the example, just switching it out with the current Jaeger setup we're using, I get this exception on all requests:
```python
ERROR:root:tracer.extract() failed
 Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/grpc_opentracing/_server.py", line 117, in _start_span
     opentracing.Format.HTTP_HEADERS, dict(metadata))
   File "/usr/local/lib/python3.6/site-packages/ddtrace/opentracer/tracer.py", line 291, in extract
     return propagator.extract(carrier)
   File "/usr/local/lib/python3.6/site-packages/ddtrace/opentracer/propagation/http.py", line 73, in extract
     raise SpanContextCorruptedException('failed to extract span context')
 opentracing.propagation.SpanContextCorruptedException: failed to extract span context
```
The spans do make it do datadog though, they're just errors of course:
![screenshot from 2018-09-16 01 48 50](https://user-images.githubusercontent.com/9902896/45591422-be8a6d80-b952-11e8-95f3-271fe5130fd8.png)

Using Python 3.6.6 with ddtrace[opentracing] and grpcio-opentracing 1.1.3, running the apps with `ddtrace-run`.



Discussion:
The cause of the corrupt span context is missing datadog tracing context from the client, of course as I soon found out. However, when the client is not instrumented - somehow child spans for redis and postgres get assigned the correct span `parent_id` as seen here below. Of course this situation is not viable because you want instrumented client calls for distributed tracing.
```python
DEBUG:ddtrace.tracer:
      name /Coins/GetPrice
        id 3041379309127771459
  trace_id 17163146044931700539
 parent_id None
   service coins
  resource /Coins/GetPrice
      type None
     start 1537131825.8142114
       end 1537131825.8169916
  duration 0.002780s
     error 1
      tags 
           component:grpc
           error:1
           error.type:failed to extract span context
           peer.ipv4:10.44.8.114
           peer.port:46348
           span.kind:server
           system.pid:1
DEBUG:ddtrace.tracer:
      name redis.command
        id 5647302167695458874
  trace_id 17163146044931700539
 parent_id 3041379309127771459
   service redis
  resource WATCH function__get_price_94d335cb22cc64fa77bb7bc6eba8b6cf_age
      type redis
     start 1537131825.8144538
       end 1537131825.8150167
  duration 0.000563s
     error 0
      tags 
           out.host:10.0.0.3
           out.port:6379
           out.redis_db:2
           redis.raw_command:WATCH function__get_price_94d335cb22cc64fa77bb7bc6eba8b6cf_age
```
![screenshot from 2018-09-16 23 17 05](https://user-images.githubusercontent.com/9902896/45601049-af64f780-ba06-11e8-8e16-0327229900f3.png)

Now, the real issue appears when the client *is* properly instrumented and sends its own spans, the server fails to assign a `parent_id` to its redis and pyscopg2 calls, as seen here:
```python
DEBUG:ddtrace.tracer:
      name /Coins/GetPrice
        id 698845531390496117
  trace_id 10364865598887188187
 parent_id 13746863732378892919
   service coins
  resource /Coins/GetPrice
      type None
     start 1537130500.7035654
       end 1537130500.709278
  duration 0.005713s
     error 0
      tags 
           component:grpc
           peer.ipv4:10.44.2.28
           peer.port:56250
           span.kind:server
           system.pid:1
DEBUG:ddtrace.tracer:writing 1 spans (enabled:True)
DEBUG:ddtrace.tracer:
      name redis.command
        id 16361053656330618518
  trace_id 14552317643450300484
 parent_id None
   service redis
  resource WATCH function__get_price_94d335cb22cc64fa77bb7bc6eba8b6cf_age
      type redis
     start 1537130500.7545977
       end 1537130500.755274
  duration 0.000676s
     error 0
      tags 
           out.host:10.0.0.3
           out.port:6379
           out.redis_db:2
           redis.raw_command:WATCH function__get_price_94d335cb22cc64fa77bb7bc6eba8b6cf_age
           system.pid:1
DEBUG:ddtrace.tracer:writing 1 spans (enabled:True)
```
![screenshot from 2018-09-16 23 20 49](https://user-images.githubusercontent.com/9902896/45601079-2b5f3f80-ba07-11e8-98a2-bd281b79d826.png)
What I'm actually expecting to see is the combination of these two screenshots, with client, server and redis calls.
Hey @raybotha,

Thanks for the detailed report. I'm investigating this right now.

Just to make sure (for both you and I): 

- both the client and the server are instrumented using the datadog opentracer
- your `tracer_init` function (or equivalent) is calling `set_global_tracer` (setting the global tracer is needed for the opentracer to interop with the datadog tracer and determine the parenting)
- you aren't getting the `SpanContextCorruptedException` when both the client and server are instrumented
- when you do instrument both the client and the server the parenting on the server is not working correctly

On a side note: since this is a distributed trace you will want to include `priority_sampling: True` in the configuration passed to the opentracer. This will prevent parts of the trace from being sampled. (We'll make this more clear in the documentation for distributed scenarios).
Hi @Kyle-Verhoog 

> both the client and the server are instrumented using the datadog opentracer

- Both the client and server are instrumented with datadog opentracer. When they are instrumented, I don't get the `SpanContextCorruptedException` exception, so I think that part is solved. When the client was not instrumented, that is when I get the first log and screenshot in my comment above - but - this was somehow when the server properly handled parenting of the redis and psycopg spans.

> your tracer_init function (or equivalent) is calling set_global_tracer (setting the global tracer is needed for the opentracer to interop with the datadog tracer and determine the parenting)

- This is my code for tracer init, the `self.tracer` is then used elsewhere for opentracing:
```python
from ddtrace.opentracer import Tracer, set_global_tracer
config = {
     'agent_hostname': HOSTNAME,
     'agent_port': 8126,
     'priority_sampling': True
}
self.tracer = Tracer(service_name, config=config)
set_global_tracer(self.tracer)
```

> you aren't getting the SpanContextCorruptedException when both the client and server are instrumented

- Yes that's right (bullet one), this is when I get the second log and screenshot in my comment above.

> when you do instrument both the client and the server the parenting on the server is not working correctly

- Yes, when both the client and server are instrumented, that is when the parenting on the server is not working properly, in this case the child spans are redis and psycopg (regular ddtrace patched), and the parent span would be an opentracing span, I'm not doing any manual tracing or assigning spans or IDs though.

> On a side note: since this is a distributed trace you will want to include priority_sampling: True in the configuration passed to the opentracer. This will prevent parts of the trace from being sampled. (We'll make this more clear in the documentation for distributed scenarios).

Regarding the side note on distributed tracing, I have set the `priority_sampling: True` as you can see above, and also set it True in my standard datadog tracer configure. I've switched to running this first-thing in my code instead of `ddtrace-run` but it doesn't make a difference to anything I believe:
```python
from ddtrace import tracer, patch_all
tracer.configure(priority_sampling=True, hostname=HOSTNAME)
patch_all()
```
Also on the side note, some spans are still sampled down with the priority_sampling enabled, these logs pop up occasionally:
```python
{"date": "2018-09-17T15:57:13.657812Z", "level": "INFO", "message": "initialized RateSampler, sample 29.605142812253128% of traces", "service": "trade_pipeline"}
{"date": "2018-09-17T15:57:13.658136Z", "level": "INFO", "message": "initialized RateSampler, sample 62.13498655279618% of traces", "service": "trade_pipeline"}
```
Any ideas on how to avoid that?
Thanks for the help.
@raybotha 

I'll look into the parenting issue, it might be related to the `grpcio-opentracing`.


As for the side note:

The logs are probably coming as a result of our agent setting the sample rate for your services. This can overridden to force the trace to be sampled in the opentracer by setting the `SAMPLING_PRIORITY` tag on the span to one of these values: https://github.com/DataDog/dd-trace-py/blob/3a4e3fe859c889da997c9ca612714eaefb1a0f4a/ddtrace/ext/priority.py#L17-L24.

The same principles as stated here: http://pypi.datadoghq.com/trace/docs/advanced_usage.html#priority-sampling apply to the opentracer as well.

Let me know if this is unclear or if that issue still persists! 😄
Thanks Kyle, I'll take a stab at setting the sampling priority on the context.

I'll also try and take a look at the interaction between datadog and the `grpcio-opentracing`, it's an odd bug.
Regarding the side note, I tried assigning a sampling priority of 2 to all spans, and also tried patching the RateSampler to sample 100%. The effect is that there are _more_ traces coming through but I think some still get discarded due to the volume, this also increases the amount of spans missing children and parents, probably due to the other spans not being sampled on a different host.

So I've gone back to the regular `priority_sampling: True` config, but I would still like to eliminate spans that get dropped out of a distributed trace. If I understand the tracer code correctly this isn't supposed to happen by the way the `Context` is used, maybe it's related to the context issue with redis/psycopg? But with this `priority_sampling` config these opentracer spans are only missing around 50% of the time with a one client and one server setup.

Resolving patch:
diff --git a/ddtrace/opentracer/tracer.py b/ddtrace/opentracer/tracer.py
--- a/ddtrace/opentracer/tracer.py
+++ b/ddtrace/opentracer/tracer.py
@@ -288,4 +288,9 @@ def extract(self, format, carrier):
         if propagator is None:
             raise opentracing.UnsupportedFormatException
 
-        return propagator.extract(carrier)
+        # we have to manually activate the returned context from a distributed
+        # trace
+        ot_span_ctx = propagator.extract(carrier)
+        dd_span_ctx = ot_span_ctx._dd_context
+        self._dd_tracer.context_provider.activate(dd_span_ctx)
+        return ot_span_ctx


Test patch:
diff --git a/tests/opentracer/test_dd_compatibility.py b/tests/opentracer/test_dd_compatibility.py
--- a/tests/opentracer/test_dd_compatibility.py
+++ b/tests/opentracer/test_dd_compatibility.py
@@ -1,5 +1,8 @@
 import ddtrace
 import opentracing
+from opentracing import Format
+
+from ddtrace.opentracer.span_context import SpanContext
 
 from tests.opentracer.utils import ot_tracer_factory, ot_tracer, dd_tracer, writer, global_tracer
 
@@ -166,3 +169,22 @@ def fn():
         assert spans[0].parent_id is None
         assert spans[1].parent_id is spans[0].span_id
         assert spans[2].parent_id is spans[1].span_id
+
+    def test_distributed_trace_propagation(self, ot_tracer, dd_tracer, writer):
+        """Ensure that a propagated span context is properly activated."""
+        span_ctx = SpanContext(trace_id=123, span_id=456)
+        carrier = {}
+        ot_tracer.inject(span_ctx, Format.HTTP_HEADERS, carrier)
+
+        # extract should activate the span so that a subsequent start_span
+        # will inherit from the propagated span context
+        ext_span_ctx = ot_tracer.extract(Format.HTTP_HEADERS, carrier)
+
+        with dd_tracer.trace('test') as span:
+            pass
+
+        assert span.parent_id == 456
+        assert span.trace_id == 123
+
+        spans = writer.pop()
+        assert len(spans) == 1