Repository: DataDog/dd-trace-py

Issue:
v0.27.0 destroys the performance of our app
Hello,

when we upgraded from v0.26.0 to v0.27.0 the performance of our Django + Mariadb + elasticsearch application went down.
Requests via the web interface took 3 times as long, often causing gateway timeouts.
Reverting to v0.26.0 of ddtrace resolves the issue.

Any idea on what might be going on here?

Greetings,

Jan


Discussion:
Not from the top of my head. There were a few changes in 0.27.0 but it's hard to see which one would cause that. If there's any chance you can bisect which particular commit introduces this regression that would help a lot. Or even basic profiling of a request that took too long would give us a few hints.
@jgb can you send me an email from your work email, brett.langdon@datadoghq.com

I'd like to gather some more details from you, but they are specific to your organization so we probably don't want to discuss here :)
Hello everybody :)

I reproduced the issue in local and it is most likely related to 0.27.0

The first difference is we get an error in 0.26.0:

```
ERROR failed_to_send traces to Datadog Agent: HTTP error status 413, reason Request Entity Too Large, message Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 22 Jul 2019 09:00:02 GMT
Content-Length: 18
```

I assume this big request is now being accepted in 0.27 and is causing the slow down.

Could be related to https://github.com/DataDog/dd-trace-py/commit/40c0eb8ee9a490b7c1cd304333ecffd60700ace2

Still digging :male_detective: 

Yeah that's something we fixed in 0.27.0. I imagine that if the payload is now accepted with 0.27..0, your local agent might be doing more processing. Did you check its CPU usage, just in case?
It looks like the agent is doing fine (no observable CPU usage yet).
The client (a Django app) is using one core at 100% for the duration of the request (150 seconds in 0.27.0).


0.27.0 is doing a lot in context.py (close_span & get)
![image](https://user-images.githubusercontent.com/1983087/61629003-0de0af00-ac84-11e9-80f1-6348fc981c78.png)


This could be due to 0252795bdfd1ee23c6fd0a6eb2e1a520f87c665c. Do you have the same profiling with 0.26.0?
@jd Can you give me your email?
I'll send you both .svg (and .prof if you need) files.

Resolving patch:
diff --git a/ddtrace/context.py b/ddtrace/context.py
--- a/ddtrace/context.py
+++ b/ddtrace/context.py
@@ -35,6 +35,7 @@ def __init__(self, trace_id=None, span_id=None, sampling_priority=None, _dd_orig
         :param int span_id: span_id of parent span
         """
         self._trace = []
+        self._finished_spans = 0
         self._current_span = None
         self._lock = threading.Lock()
 
@@ -126,6 +127,7 @@ def close_span(self, span):
         cycles inside _trace list.
         """
         with self._lock:
+            self._finished_spans += 1
             self._set_current_span(span._parent)
 
             # notify if the trace is not closed properly; this check is executed only
@@ -136,12 +138,13 @@ def close_span(self, span):
             # some children. On the other hand, asynchronous web frameworks still expect
             # to close the root span after all the children.
             tracer = getattr(span, '_tracer', None)
-            unfinished_spans = [x for x in self._trace if not x._finished]
-            if tracer and tracer.debug_logging and span._parent is None and unfinished_spans:
-                log.debug('Root span "%s" closed, but the trace has %d unfinished spans:',
-                          span.name, len(unfinished_spans))
-                for wrong_span in unfinished_spans:
-                    log.debug('\n%s', wrong_span.pprint())
+            if tracer and tracer.debug_logging and span._parent is None:
+                unfinished_spans = [x for x in self._trace if not x._finished]
+                if unfinished_spans:
+                    log.debug('Root span "%s" closed, but the trace has %d unfinished spans:',
+                              span.name, len(unfinished_spans))
+                    for wrong_span in unfinished_spans:
+                        log.debug('\n%s', wrong_span.pprint())
 
     def _is_sampled(self):
         return any(span.sampled for span in self._trace)
@@ -156,9 +159,8 @@ def get(self):
         This operation is thread-safe.
         """
         with self._lock:
-            finished_spans = [t for t in self._trace if t._finished]
             # All spans are finished?
-            if len(finished_spans) == len(self._trace):
+            if self._finished_spans == len(self._trace):
                 # get the trace
                 trace = self._trace
                 sampled = self._is_sampled()
@@ -178,36 +180,40 @@ def get(self):
 
                 # clean the current state
                 self._trace = []
+                self._finished_spans = 0
                 self._parent_trace_id = None
                 self._parent_span_id = None
                 self._sampling_priority = None
                 return trace, sampled
 
-            elif self._partial_flush_enabled and len(finished_spans) >= self._partial_flush_min_spans:
-                # partial flush when enabled and we have more than the minimal required spans
-                trace = self._trace
-                sampled = self._is_sampled()
-                sampling_priority = self._sampling_priority
-                # attach the sampling priority to the context root span
-                if sampled and sampling_priority is not None and trace:
-                    trace[0].set_metric(SAMPLING_PRIORITY_KEY, sampling_priority)
-                origin = self._dd_origin
-                # attach the origin to the root span tag
-                if sampled and origin is not None and trace:
-                    trace[0].set_tag(ORIGIN_KEY, origin)
-
-                # Set hostname tag if they requested it
-                if config.report_hostname:
-                    # DEV: `get_hostname()` value is cached
-                    trace[0].set_tag(HOSTNAME_KEY, hostname.get_hostname())
-
-                # Any open spans will remain as `self._trace`
-                # Any finished spans will get returned to be flushed
-                self._trace = [t for t in self._trace if not t._finished]
-
-                return finished_spans, sampled
-            else:
-                return None, None
+            elif self._partial_flush_enabled:
+                finished_spans = [t for t in self._trace if t._finished]
+                if len(finished_spans) >= self._partial_flush_min_spans:
+                    # partial flush when enabled and we have more than the minimal required spans
+                    trace = self._trace
+                    sampled = self._is_sampled()
+                    sampling_priority = self._sampling_priority
+                    # attach the sampling priority to the context root span
+                    if sampled and sampling_priority is not None and trace:
+                        trace[0].set_metric(SAMPLING_PRIORITY_KEY, sampling_priority)
+                    origin = self._dd_origin
+                    # attach the origin to the root span tag
+                    if sampled and origin is not None and trace:
+                        trace[0].set_tag(ORIGIN_KEY, origin)
+
+                    # Set hostname tag if they requested it
+                    if config.report_hostname:
+                        # DEV: `get_hostname()` value is cached
+                        trace[0].set_tag(HOSTNAME_KEY, hostname.get_hostname())
+
+                    self._finished_spans = 0
+
+                    # Any open spans will remain as `self._trace`
+                    # Any finished spans will get returned to be flushed
+                    self._trace = [t for t in self._trace if not t._finished]
+
+                    return finished_spans, sampled
+            return None, None
 
 
 class ThreadLocalContext(object):


Test patch: