Repository: DataDog/dd-trace-py

Issue:
`ddtrace>=0.30.0` starts a thread at import, leading to issues with gevent.monkey.patch_all
# Package versions

```
% python --version
Python 3.7.6
```

```
% pip freeze
ddtrace==0.33.0
gevent==1.4.0
greenlet==0.4.15
psutil==5.6.7
```

This occurs with `python>=3.6`, `gevent>=1.3` (older versions are not compatible with `py3.6` anyway) and `ddtrace>=0.30.0,<=0.33.0` (current latest as of this bug filing).

# Steps to reproduce

With `test_gevent.py` as follows:

```
import ddtrace

import gevent.monkey

gevent.monkey.patch_all()
```

run

```
% python -c "import test_gevent"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 152, in __exit__
  File "<frozen importlib._bootstrap>", line 107, in release
RuntimeError: cannot release un-acquired lock
```

The `gevent` patching fails because a thread has already been started, but `ddtrace` patching has not been explicitly invoked.

# Possible cause

The `Writer` and `Worker` classes were merged in https://github.com/DataDog/dd-trace-py/pull/988 . As part of that change, the worker thread is now started when `AgentWriter.__init__` is called, whereas (if I read correctly) previously the worker thread was only started upon the first call to `AgentWriter.write`:

Change: https://github.com/DataDog/dd-trace-py/commit/217a8c62461ec742667306d4974fca1cb459db68#diff-88c4dee1a45666e54351f6960196a037R36 (more obvious in split view, but previously `_reset_worker` started the worker thread in the `write` function)

We will work around this by only importing `ddtrace` if we are certain we will then call `ddtrace.patch`, but ideally simply importing `ddtrace` would be a safe operation.


Discussion:

Resolving patch:
diff --git a/ddtrace/_worker.py b/ddtrace/_worker.py
--- a/ddtrace/_worker.py
+++ b/ddtrace/_worker.py
@@ -32,13 +32,14 @@ def __init__(self, interval=_DEFAULT_INTERVAL, exit_timeout=None, name=None, dae
         self._thread = threading.Thread(target=self._target, name=name)
         self._thread.daemon = daemon
         self._stop = threading.Event()
+        self.started = False
         self.interval = interval
         self.exit_timeout = exit_timeout
         atexit.register(self._atexit)
 
     def _atexit(self):
         self.stop()
-        if self.exit_timeout is not None:
+        if self.exit_timeout is not None and self.started:
             key = 'ctrl-break' if os.name == 'nt' else 'ctrl-c'
             _LOG.debug(
                 'Waiting %d seconds for %s to finish. Hit %s to quit.',
@@ -50,6 +51,7 @@ def start(self):
         """Start the periodic worker."""
         _LOG.debug('Starting %s thread', self._thread.name)
         self._thread.start()
+        self.started = True
 
     def stop(self):
         """Stop the worker."""
diff --git a/ddtrace/internal/writer.py b/ddtrace/internal/writer.py
--- a/ddtrace/internal/writer.py
+++ b/ddtrace/internal/writer.py
@@ -50,7 +50,7 @@ def __init__(
         )
         if hasattr(time, "thread_time"):
             self._last_thread_time = time.thread_time()
-        self.start()
+        self._started = False
 
     def recreate(self):
         """ Create a new instance of :class:`AgentWriter` using the same settings from this instance
@@ -76,6 +76,12 @@ def _send_stats(self):
         return bool(config.health_metrics_enabled and self.dogstatsd)
 
     def write(self, spans=None, services=None):
+        # Start the AgentWriter on first write.
+        # Starting it earlier might be an issue with gevent, see:
+        # https://github.com/DataDog/dd-trace-py/issues/1192
+        if self._started is False:
+            self.start()
+            self._started = True
         if spans:
             self._trace_queue.put(spans)
 


Test patch:
diff --git a/tests/test_tracer.py b/tests/test_tracer.py
--- a/tests/test_tracer.py
+++ b/tests/test_tracer.py
@@ -629,11 +629,6 @@ def task(t, errors):
                 assert t.writer != original_writer
                 assert t.writer._trace_queue != original_writer._trace_queue
 
-            # Stop the background worker so we don't accidetnally flush the
-            # queue before we can assert on it
-            t.writer.stop()
-            t.writer.join()
-
         # Assert the trace got written into the correct queue
         assert original_writer._trace_queue.qsize() == 0
         assert t.writer._trace_queue.qsize() == 1
@@ -656,11 +651,6 @@ def task(t, errors):
         assert t.writer == original_writer
         assert t.writer._trace_queue == original_writer._trace_queue
 
-        # Stop the background worker so we don't accidentally flush the
-        # queue before we can assert on it
-        t.writer.stop()
-        t.writer.join()
-
     # Assert the trace got written into the correct queue
     assert original_writer._trace_queue.qsize() == 1
     assert t.writer._trace_queue.qsize() == 1
diff --git a/tests/test_worker.py b/tests/test_worker.py
--- a/tests/test_worker.py
+++ b/tests/test_worker.py
@@ -56,3 +56,16 @@ def test_restart():
 
     with pytest.raises(RuntimeError):
         w.start()
+
+
+def test_atexit():
+    w = _worker.PeriodicWorkerThread(exit_timeout=1)
+    w.start()
+    assert w.started
+    w._atexit()
+
+
+def test_atexit_not_started():
+    w = _worker.PeriodicWorkerThread(exit_timeout=1)
+    assert not w.started
+    w._atexit()