Repository: DataDog/dd-trace-py

Issue:
Stuck in "Waiting for GIL"
Hi,

We found a thread that seems blocked forever:

```
gdb python 30107
```

Here's the single thread:

```
(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7f1dd852e4c0 (LWP 30107) "/opt/simon/rele" 0x00007f1dd7d2e146 in do_futex_wait.constprop () from /lib64/libpthread.so.0
```

Here's the backtrace:

```
raceback (most recent call first):
  Waiting for the GIL
  File "/opt/simon/releases/b038662ce7d140609b1e4883a3ea0edf77851537/venv/lib/python2.7/site-packages/ddtrace/sampler.py", line 88, in sample
    with self._lock:
  File "/opt/simon/releases/b038662ce7d140609b1e4883a3ea0edf77851537/venv/lib/python2.7/site-packages/ddtrace/tracer.py", line 251, in start_span
    if self.priority_sampler.sample(span):
  File "/opt/simon/releases/b038662ce7d140609b1e4883a3ea0edf77851537/venv/lib/python2.7/site-packages/ddtrace/tracer.py", line 376, in trace
    span_type=span_type,
  File "/opt/simon/releases/b038662ce7d140609b1e4883a3ea0edf77851537/venv/lib/python2.7/site-packages/ddtrace/contrib/django/cache.py", line 56, in wrapped
    with tracer.trace('django.cache', span_type=TYPE, service=cache_service_name) as span:
```

I'm not sure how to troubleshoot from here.  Have you seen this before, or have any suggestions on what I could look at?  It's the only thread in process so nothing should be holding the GIL.


Discussion:
My boss has been digging into this, some more useful info:

```
what we see is sockets open to https that hang forever because we don't set timeouts
ddtrace is instantiating that sampler in the parent
then we're forking children
each child is then using the lock when it samples
and those sometimes hang
and we use billiard not multiprocessing if that helps
```

So hopefully that helps, ddtrace samples the parent, we generate children, and these sometimes hang.  We confirmed that's true by doing a "return/finish" in gdb to force them to carry on.
Are the parent or the children blocked, or both?
Since the app is forking, the parent/children relation should have no impact on the lock since they are duplicated.
Just the child, and only 1 of multiple.  The others had all exited cleanly, with just the 1 child left waiting for the lock.
After further experimentation, we have noticed this ONLY happens in memcache.  We've excluded tracing for that and seen no hangs since.

We're using an ancient django (1.8).  Could we be seeing some weird behavior with ddtrace that is fixed on a newer django, I wonder.
We turned tracing back on with memcache disabled and while it was much better we eventually saw it in other parts of the code, a few times around mysql.

I saw cases where there was "waiting for GIL lock" in a process that only had one thread.  Very confusing. :(  We've disabled tracing in general again.
> I saw cases where there was "waiting for GIL lock" in a process that only had one thread. Very confusing. :( We've disabled tracing in general again.

Do you have the backtrace of such a process? That'd be utterly interesting to see where it hangs, and if it's always the same place. 

For now we only have the backtrace you provided in https://github.com/DataDog/dd-trace-py/issues/935#issue-443463736 and we'd like to be sure it's always the same code path that is deadlocking.
Last one I have was:

```Traceback (most recent call first):
  Waiting for the GIL
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/ddtrace/sampler.py", line 88, in sample
    with self._lock:
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/ddtrace/tracer.py", line 251, in start_span
    if self.priority_sampler.sample(span):
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/ddtrace/tracer.py", line 376, in trace
    span_type=span_type,
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/ddtrace/contrib/dbapi/__init__.py", line 46, in _trace_method
    with pin.tracer.trace(name, service=service, resource=resource) as s:
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/ddtrace/contrib/dbapi/__init__.py", line 91, in execute
    return self._trace_method(self.__wrapped__.execute, self._self_datadog_name, query, {}, query, *args, **kwargs)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 287, in init_connection_state
    cursor.execute('SET SQL_AUTO_IS_NULL = 0')
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 121, in connect
    self.init_connection_state()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 135, in _cursor
    self.ensure_connection()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 164, in cursor
    cursor = self.make_cursor(self._cursor())
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 838, in execute_sql
    cursor = self.connection.cursor()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 95, in __getstate__
    self._fetch_all()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib64/python2.7/copy_reg.py", line 84, in _reduce_ex
    dict = getstate()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 135, in __reduce__
    return super(QuerySet, self).__reduce__()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/memcache.py", line 782, in _val_to_store_info
    pickler.dump(val)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/memcache.py", line 812, in _unsafe_set
    store_info = self._val_to_store_info(val, min_compress_len)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/memcache.py", line 835, in _set
    return _unsafe_set()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/memcache.py", line 584, in set
    return self._set("set", key, val, time, min_compress_len)

File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 91, in set
    if not self._cache.set(key, value, self.get_backend_timeout(timeout)):
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/customers/blacklist.py", line 55, in fetch_blacklist
    cache.set(self._cache_key, val, CACHE_TTL)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/customers/blacklist.py", line 33, in __init__
    blacklist = self.fetch_blacklist()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/flows/ops.py", line 721, in _blacklist
    blacklist_lookup = CustomerBlacklistLookup(self.organization)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/flows/ops.py", line 540, in _process_contacts
    contacts = self._blacklist(contacts)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/flows/ops.py", line 339, in do_work
    processed_batch = self._process_contacts(batch)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/queue.py", line 199, in worker
    result = task_queue.do_work(worker_index, item)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/billiard/process.py", line 123, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap
    self.run()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/billiard/forking.py", line 105, in __init__
    code = process_obj._bootstrap()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/billiard/process.py", line 137, in start
    self._popen = Popen(self)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/queue.py", line 268, in start_procs
    p.start()
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/flows/ops.py", line 489, in start_procs
    num_procs=self.num_procs(contact_count)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/flows/sync.py", line 613, in trigger
    task_queue.start_procs(op_contact_count)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/campaigns/export_stage_hook.py", line 32, in wrapper
    context = func(*args, **kwargs)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/flows/sync.py", line 364, in _export
    OperationTaskQueue.REMOVE_OP)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/flows/sync.py", line 147, in export
    return self._export(sync)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/event_log/tools.py", line 40, in wrap
    result = f(self, *ags, **kw)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/simon/management/commands/incremental_sync.py", line 68, in handle
    executor.export(campaign_sync)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/simon/releases/34641b8a63b60644e67ba0f310d22084096af63c/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()```
One before that:

```
Traceback (most recent call first):
  Waiting for the GIL
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/ddtrace/sampler.py", line 88, in sample
    with self._lock:
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/ddtrace/tracer.py", line 251, in start_span
    if self.priority_sampler.sample(span):
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/ddtrace/tracer.py", line 376, in trace
    span_type=span_type,
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/ddtrace/contrib/dbapi/__init__.py", line 46, in _trace_method
    with pin.tracer.trace(name, service=service, resource=resource) as s:
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/ddtrace/contrib/dbapi/__init__.py", line 91, in execute
    return self._trace_method(self.__wrapped__.execute, self._self_datadog_name, query, {}, query, *args, **kwargs)
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 287, in init_connection_state
    cursor.execute('SET SQL_AUTO_IS_NULL = 0')
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 121, in connect
    self.init_connection_state()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 135, in _cursor
    self.ensure_connection()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 164, in cursor
    cursor = self.make_cursor(self._cursor())
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 838, in execute_sql
    cursor = self.connection.cursor()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 95, in __getstate__
    self._fetch_all()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib64/python2.7/copy_reg.py", line 84, in _reduce_ex
    dict = getstate()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/db/models/query.py", line 135, in __reduce__
    return super(QuerySet, self).__reduce__()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/memcache.py", line 782, in _val_to_store_info
    pickler.dump(val)
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/memcache.py", line 812, in _unsafe_set
    store_info = self._val_to_store_info(val, min_compress_len)
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/memcache.py", line 835, in _set
    return _unsafe_set()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/memcache.py", line 584, in set
    return self._set("set", key, val, time, min_compress_len)
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/venv/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 91, in set
    if not self._cache.set(key, value, self.get_backend_timeout(timeout)):
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/simon/customers/blacklist.py", line 55, in fetch_blacklist
    cache.set(self._cache_key, val, CACHE_TTL)
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/simon/customers/blacklist.py", line 33, in __init__
    blacklist = self.fetch_blacklist()
  File "/opt/simon/releases/d14859bdd2e1c84f8976b928f39a516f436f2a6c/simon/flows/ops.py", line 722, in _blacklist
    blacklist_lookup = CustomerBlacklistLookup(self.organization)
```

Resolving patch:
diff --git a/ddtrace/sampler.py b/ddtrace/sampler.py
--- a/ddtrace/sampler.py
+++ b/ddtrace/sampler.py
@@ -2,8 +2,6 @@
 
 Any `sampled = False` trace won't be written, and can be ignored by the instrumentation.
 """
-from threading import Lock
-
 from .compat import iteritems
 from .internal.logger import get_logger
 
@@ -50,15 +48,6 @@ def sample(self, span):
         return sampled
 
 
-def _key(service=None, env=None):
-    service = service or ''
-    env = env or ''
-    return 'service:' + service + ',env:' + env
-
-
-_default_key = _key()
-
-
 class RateByServiceSampler(object):
     """Sampler based on a rate, by service
 
@@ -66,34 +55,40 @@ class RateByServiceSampler(object):
     The sample rate is kept independently for each service/env tuple.
     """
 
+    @staticmethod
+    def _key(service=None, env=None):
+        """Compute a key with the same format used by the Datadog agent API."""
+        service = service or ''
+        env = env or ''
+        return 'service:' + service + ',env:' + env
+
     def __init__(self, sample_rate=1):
-        self._lock = Lock()
-        self._by_service_samplers = {}
-        self._by_service_samplers[_default_key] = RateSampler(sample_rate)
+        self.sample_rate = sample_rate
+        self._by_service_samplers = self._get_new_by_service_sampler()
 
-    def _set_sample_rate_by_key(self, sample_rate, key):
-        with self._lock:
-            if key in self._by_service_samplers:
-                self._by_service_samplers[key].set_sample_rate(sample_rate)
-            else:
-                self._by_service_samplers[key] = RateSampler(sample_rate)
+    def _get_new_by_service_sampler(self):
+        return {
+            self._default_key: RateSampler(self.sample_rate)
+        }
 
     def set_sample_rate(self, sample_rate, service='', env=''):
-        self._set_sample_rate_by_key(sample_rate, _key(service, env))
+        self._by_service_samplers[self._key(service, env)] = RateSampler(sample_rate)
 
     def sample(self, span):
         tags = span.tracer().tags
         env = tags['env'] if 'env' in tags else None
-        key = _key(span.service, env)
-        with self._lock:
-            if key in self._by_service_samplers:
-                return self._by_service_samplers[key].sample(span)
-            return self._by_service_samplers[_default_key].sample(span)
+        key = self._key(span.service, env)
+        return self._by_service_samplers.get(
+            key, self._by_service_samplers[self._default_key]
+        ).sample(span)
 
     def set_sample_rate_by_service(self, rate_by_service):
+        new_by_service_samplers = self._get_new_by_service_sampler()
         for key, sample_rate in iteritems(rate_by_service):
-            self._set_sample_rate_by_key(sample_rate, key)
-        with self._lock:
-            for key in list(self._by_service_samplers):
-                if key not in rate_by_service and key != _default_key:
-                    del self._by_service_samplers[key]
+            new_by_service_samplers[key] = RateSampler(sample_rate)
+
+        self._by_service_samplers = new_by_service_samplers
+
+
+# Default key for service with no specific rate
+RateByServiceSampler._default_key = RateByServiceSampler._key()


Test patch:
diff --git a/tests/test_sampler.py b/tests/test_sampler.py
--- a/tests/test_sampler.py
+++ b/tests/test_sampler.py
@@ -3,7 +3,7 @@
 import unittest
 
 from ddtrace.span import Span
-from ddtrace.sampler import RateSampler, AllSampler, _key, _default_key
+from ddtrace.sampler import RateSampler, AllSampler, RateByServiceSampler
 from ddtrace.compat import iteritems
 from tests.test_tracer import get_dummy_tracer
 from ddtrace.constants import SAMPLING_PRIORITY_KEY, SAMPLE_RATE_METRIC_KEY
@@ -56,14 +56,16 @@ def test_deterministic_behavior(self):
 
 class RateByServiceSamplerTest(unittest.TestCase):
     def test_default_key(self):
-        assert 'service:,env:' == _default_key, 'default key should correspond to no service and no env'
+        assert (
+            'service:,env:' == RateByServiceSampler._default_key
+        ), 'default key should correspond to no service and no env'
 
     def test_key(self):
-        assert _default_key == _key()
-        assert 'service:mcnulty,env:' == _key(service='mcnulty')
-        assert 'service:,env:test' == _key(env='test')
-        assert 'service:mcnulty,env:test' == _key(service='mcnulty', env='test')
-        assert 'service:mcnulty,env:test' == _key('mcnulty', 'test')
+        assert RateByServiceSampler._default_key == RateByServiceSampler._key()
+        assert 'service:mcnulty,env:' == RateByServiceSampler._key(service='mcnulty')
+        assert 'service:,env:test' == RateByServiceSampler._key(env='test')
+        assert 'service:mcnulty,env:test' == RateByServiceSampler._key(service='mcnulty', env='test')
+        assert 'service:mcnulty,env:test' == RateByServiceSampler._key('mcnulty', 'test')
 
     def test_sample_rate_deviation(self):
         for sample_rate in [0.1, 0.25, 0.5, 1]: