Repository: DataDog/dd-trace-py

Issue:
nginx-opentracing + libdd_opentracing_plugin: SpanContextCorruptedException: failed to extract span context
I'm trying to set up an integration of nginx + nginx-opentracing module + DataDog tracer plugin + sample python app in order to get working multi-span traces in a manner when an app uses propagated context.

I'm getting the following error on every call:

```
ERROR:root:trace extract failed: failed to extract span context
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/opentracing_instrumentation/http_server.py", line 75, in before_request
    format=Format.HTTP_HEADERS, carrier=carrier
  File "/usr/local/lib/python2.7/dist-packages/ddtrace-0.20.4-py2.7.egg/ddtrace/opentracer/tracer.py", line 294, in extract
    ot_span_ctx = propagator.extract(carrier)
  File "/usr/local/lib/python2.7/dist-packages/ddtrace-0.20.4-py2.7.egg/ddtrace/opentracer/propagation/http.py", line 73, in extract
    raise SpanContextCorruptedException('failed to extract span context')
SpanContextCorruptedException: failed to extract span context
```

Components used:
 - nginx/1.15.7
 - nginx-opentracing: 
https://github.com/opentracing-contrib/nginx-opentracing/releases/tag/v0.8.0
 - DataDog tracer plugin: https://github.com/DataDog/dd-opentracing-cpp/releases/download/v0.4.2/linux-amd64-libdd_opentracing_plugin.so.gz

nginx configuration:

```
# configuration file /etc/nginx/nginx.conf:
load_module modules/ngx_http_opentracing_module.so;

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" $request_id';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    opentracing_load_tracer /etc/nginx/linux-amd64-libdd_opentracing_plugin.so /etc/nginx/dd-config.json;
    opentracing on;
    opentracing_trace_locations off;
    opentracing_tag http_user_agent $http_user_agent;
    opentracing_tag http_uri $request_uri;
    opentracing_tag http_request_id $request_id;

    include /etc/nginx/conf.d/*.conf;
}

# configuration file /etc/nginx/conf.d/default.conf:
upstream u {
    server 62.210.92.35:80;
    keepalive 20;
    zone u 128k;
}

upstream upload-app {
    server 127.0.0.1:8080;
}

server {
    listen       80 default_server;
    server_name  localhost;

    opentracing_operation_name $uri;

    location / {
        opentracing_propagate_context;
        proxy_set_header Host nginx.org;
        proxy_set_header Connection "";
        proxy_http_version 1.1;
        proxy_pass http://u;
    }

    location /upload/ {
        opentracing_propagate_context;
        proxy_pass http://upload-app;
        client_max_body_size 256m;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}
```

DataDog tracer configuration (/etc/nginx/dd-config.json):

```
{
  "service": "nginx",
  "operation_name_override": "nginx.handle",
  "agent_host": "localhost",
  "agent_port": 8126
}
```

DataDog agent version, OS used:

```
# dpkg -s datadog-agent
Package: datadog-agent
Status: install ok installed
Priority: extra
Section: utils
Installed-Size: 390206
Maintainer: Datadog Packages <package@datadoghq.com>
Architecture: amd64
Version: 1:6.9.0-1
Description: Datadog Monitoring Agent
  The Datadog Monitoring Agent is a lightweight process that monitors system
  processes and services, and sends information back to your Datadog account.
  .
  This package installs and runs the advanced Agent daemon, which queues and
  forwards metrics from your applications as well as system services.
  .
  See http://www.datadoghq.com/ for more information
License: Apache License Version 2.0
Vendor: Datadog <package@datadoghq.com>
Homepage: http://www.datadoghq.com

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic
```

App itself:

```
#!/usr/bin/env python

import logging
import hashlib
from flask import Flask
from flask import request
from  werkzeug.debug import get_current_traceback
from opentracing_instrumentation import http_server
from opentracing_instrumentation import config
import opentracing
from ddtrace.opentracer import Tracer
from random import randint

application = Flask(__name__)
tracer = None 

def init_dd_tracer(service_name='upload-app'):
    print "INIT DATADOG TRACER"
    return Tracer(service_name=service_name, config={})

@application.before_request
def before_request():
    global tracer
    request.stderr = request.environ['wsgi.errors'] if 'wsgi.errors' in request.environ else stderr
    headers_summary = "HEADERS:\n\n" + "\n".join(["{0}: {1}".format(k, request.headers[k]) for k in sorted(request.headers.keys())]) + "\n"
    request.stderr.write(headers_summary)

    request.full_url = request.url
    request.remote_ip = request.remote_addr
    request.remote_port = request.environ['REMOTE_PORT']
    request.caller_name = "n/a"
    request.operation = request.method

    if not tracer:
        tracer = init_dd_tracer()

    request.span = http_server.before_request(request=request, tracer=tracer)

@application.route("/", methods=['GET', 'POST'])
def default():
    try:
        environ_summary = "ENVIRON:\n\n" + "\n".join(["{0}: {1}".format(k, request.environ[k]) for k in sorted(request.environ.keys())]) + "\n"
        args = "REQUEST ARGS: %s" % request.args
        body = "REQUETS BODY: %s" % request.data
        return "%s\n%s\n%s\n" % (args, body, environ_summary)
    except Exception, e:
        track = get_current_traceback(skip=1, show_hidden_frames=True, ignore_system_exceptions=False)
        track.log()
        abort(500)

@application.route("/upload/", methods=['GET', 'POST'])
@application.route("/upload-http/", methods=['GET', 'POST'])
def upload():
    global tracer
    with tracer.start_span('ProcessUpload', child_of=request.span) as span:
        span.log_kv({'ProcessUpload': 'started'})
        span.set_tag('payload-size', int(request.headers.get('Content-Length')) if 'Content-Length' in request.headers else 0)

        if 'Content-Length' in request.headers and int(request.headers.get('Content-Length')):
            body = request.stream.read()
            for x in range(1, randint(2, 10)):
                 with tracer.start_span('SubPart%02d' % x, child_of=span) as subpart_span:
                     subpart_span.log_kv({'subpart_iteration': x, 'action': 'begin'})
                     m = hashlib.md5()
                     m.update(body)
                     response_body = "%d:%s\n" % (len(body), m.hexdigest())
                     subpart_span.log_kv({'subpart_iteration': x, 'action': 'end'})
            request.stderr.write('ProcessUpload finished with %d iterations\n' % x)

        else:
            response_body = 'no data was uploaded'

        try:
            span.set_tag('iterations', x)
        except NameError:
            pass

        span.log_kv({'ProcessUpload': 'finished'})

    return response_body

@application.errorhandler(500)
def internal_error(error):
    return "500 error"

if __name__ == "__main__":
    application.debug = True
    application.config['PROPAGATE_EXCEPTIONS'] = True
    application.run(host='127.0.0.1', port=8080)
```



Discussion:

Resolving patch:
diff --git a/ddtrace/propagation/http.py b/ddtrace/propagation/http.py
--- a/ddtrace/propagation/http.py
+++ b/ddtrace/propagation/http.py
@@ -56,34 +56,35 @@ def parent_call():
             headers[HTTP_HEADER_SAMPLING_PRIORITY] = str(span_context.sampling_priority)
 
     @staticmethod
-    def extract_trace_id(headers):
-        trace_id = 0
+    def extract_header_value(possible_header_names, headers, default=None):
+        for header, value in headers.items():
+            for header_name in possible_header_names:
+                if header.lower() == header_name.lower():
+                    return value
 
-        for key in POSSIBLE_HTTP_HEADER_TRACE_IDS:
-            if key in headers:
-                trace_id = headers.get(key)
+        return default
 
-        return int(trace_id)
+    @staticmethod
+    def extract_trace_id(headers):
+        return int(
+            HTTPPropagator.extract_header_value(
+                POSSIBLE_HTTP_HEADER_TRACE_IDS, headers, default=0,
+            )
+        )
 
     @staticmethod
     def extract_parent_span_id(headers):
-        parent_span_id = 0
-
-        for key in POSSIBLE_HTTP_HEADER_PARENT_IDS:
-            if key in headers:
-                parent_span_id = headers.get(key)
-
-        return int(parent_span_id)
+        return int(
+            HTTPPropagator.extract_header_value(
+                POSSIBLE_HTTP_HEADER_PARENT_IDS, headers, default=0,
+            )
+        )
 
     @staticmethod
     def extract_sampling_priority(headers):
-        sampling_priority = None
-
-        for key in POSSIBLE_HTTP_HEADER_SAMPLING_PRIORITIES:
-            if key in headers:
-                sampling_priority = headers.get(key)
-
-        return sampling_priority
+        return HTTPPropagator.extract_header_value(
+            POSSIBLE_HTTP_HEADER_SAMPLING_PRIORITIES, headers,
+        )
 
     def extract(self, headers):
         """Extract a Context from HTTP headers into a new Context.


Test patch: