Repository: DataDog/dd-trace-py

Issue:
Profiling causing gunicorn timeout (flask app)
### Which version of dd-trace-py are you using?

```
ddtrace[profiling]==0.37.0
```

### Which version of the libraries are you using?

```
flask==1.1.2
gunicorn==20.0.4
ddtrace[profiling]==0.37.0
datadog==0.36.0
```


### How can we reproduce your problem?
I'm using a simple hello world app flask application for the demonstration of this problem but the issue is happening also when we tried to integrated the profiling with our Flask project, i thought the issue was related to our project code but in this demo proved that it's a general issue, in `wsgi.py` you can add :

```
import ddtrace.profiling.auto
```

my gunicorn config is the default "simple" one : 

```
gunicorn --bind 0.0.0.0:8000 wsgi:app
```
### What is the result that you get?

```
docker run -p 8000:8000 flask/hello-world
[2020-05-07 15:17:06 +0000] [6] [INFO] Starting gunicorn 20.0.4
[2020-05-07 15:17:06 +0000] [6] [INFO] Listening at: http://0.0.0.0:8000 (6)
[2020-05-07 15:17:06 +0000] [6] [INFO] Using worker: sync
[2020-05-07 15:17:06 +0000] [9] [INFO] Booting worker with pid: 9
[2020-05-07 15:17:52 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:9)
[2020-05-07 15:17:52 +0000] [9] [INFO] Worker exiting (pid: 9)
[2020-05-07 15:17:52 +0000] [13] [INFO] Booting worker with pid: 13
```

### What is the result that you expected?
When i removed the auto profiler the app works and got the response.
Also strangely when using the "manual" profiler 

```
from ddtrace.profiling import Profiler

prof = Profiler()
prof.start()
``` 

The app works as expected and i get profiling events in my Datadog account, so my believe is that the auto profiler causes a lock somewhere in its code resulting in Gunicorn workers timeout.



Discussion:
I'm experiencing this issue as well with `ddtrace==0.41.0`. Is there anything we can do about this?
Thanks for the report!
@youssefNM could you share your minimal test case to reproduce this? That'd be helpful.

Is pressing C-c giving you a backtrace or something of where it'd be stuck?
I've run into something similar where my flask application hangs when enabling automatic profiling.  In my case it seems to happen when all of the following are true:

* Running inside a Docker container
* Using gunicorn with gevent
* Importing a library that uses the C library *after* enabling automatic profiling

If I change any of the conditions above it works for me.

I've created a gist here with a reproducer: https://gist.github.com/russellcardullo/c7f0e16f392dc2b393b61cfbc6b490de
Thanks for the detailled report @russellcardullo!
This is only a problem with the `gevent` worker class. This happens because the `patch_all` from gevent is called too late *after* the profiler has started.

You need to use something like this in Flask for example:

```
@app.before_first_request
def start_profiler():
    import ddtrace.profiling.auto
```

That should work fine.

I'll check if we can detect this and report it and improve the documentation on our side.
This is actually not true. My workaround works, but it has nothing to do with gevent monkey patching.
It's just `find_library` that deadlock somehow on Linux (Darwin is fine) when the profiler is loaded.

I'm still digging into this.

Resolving patch:
diff --git a/ddtrace/profiling/_periodic.py b/ddtrace/profiling/_periodic.py
--- a/ddtrace/profiling/_periodic.py
+++ b/ddtrace/profiling/_periodic.py
@@ -71,6 +71,12 @@ def __init__(self, interval, target, name=None, on_shutdown=None):
         """
         super(_GeventPeriodicThread, self).__init__(interval, target, name, on_shutdown)
         self._tident = None
+        self._periodic_started = False
+        self._periodic_stopped = False
+
+    def _reset_internal_locks(self, is_alive=False):
+        # Called by Python via `threading._after_fork`
+        self._periodic_stopped = True
 
     @property
     def ident(self):
@@ -79,14 +85,22 @@ def ident(self):
     def start(self):
         """Start the thread."""
         self.quit = False
-        self.has_quit = False
+        if self._tident is not None:
+            raise RuntimeError("threads can only be started once")
         self._tident = _nogevent.start_new_thread(self.run, tuple())
         if _nogevent.threading_get_native_id:
             self._native_id = _nogevent.threading_get_native_id()
 
+        # Wait for the thread to be started to avoid race conditions
+        while not self._periodic_started:
+            time.sleep(self.SLEEP_INTERVAL)
+
+    def is_alive(self):
+        return not self._periodic_stopped and self._periodic_started
+
     def join(self, timeout=None):
         # FIXME: handle the timeout argument
-        while not self.has_quit:
+        while self.is_alive():
             time.sleep(self.SLEEP_INTERVAL)
 
     def stop(self):
@@ -97,6 +111,9 @@ def run(self):
         """Run the target function periodically."""
         # Do not use the threading._active_limbo_lock here because it's a gevent lock
         threading._active[self._tident] = self
+
+        self._periodic_started = True
+
         try:
             while self.quit is False:
                 self._target()
@@ -114,8 +131,8 @@ def run(self):
                 raise
         finally:
             try:
+                self._periodic_stopped = True
                 del threading._active[self._tident]
-                self.has_quit = True
             except Exception:
                 # Exceptions might happen during interpreter shutdown.
                 # We're mimicking what `threading.Thread` does in daemon mode, we ignore them.


Test patch:
diff --git a/tests/profiling/gevent_fork.py b/tests/profiling/gevent_fork.py
new file mode 100644
--- /dev/null
+++ b/tests/profiling/gevent_fork.py
@@ -0,0 +1,19 @@
+import os
+import sys
+
+import gevent.monkey
+
+gevent.monkey.patch_all()
+
+from ddtrace.profiling import profiler  # noqa
+
+p = profiler.Profiler().start(profile_children=True)
+
+pid = os.fork()
+if pid == 0:
+    print("Exiting")
+else:
+    print(pid)
+    pid, status = os.waitpid(pid, 0)
+    print("Exited")
+    sys.exit(os.WEXITSTATUS(status))
diff --git a/tests/profiling/test_main.py b/tests/profiling/test_main.py
--- a/tests/profiling/test_main.py
+++ b/tests/profiling/test_main.py
@@ -79,3 +79,10 @@ def test_fork(tmp_path, monkeypatch):
     child_pid = stdout.decode().strip()
     check_pprof_file(filename + "." + str(pid) + ".1")
     check_pprof_file(filename + "." + str(child_pid) + ".1")
+
+
+@pytest.mark.skipif(not os.getenv("DD_PROFILE_TEST_GEVENT", False), reason="Not testing gevent")
+def test_fork_gevent(monkeypatch):
+    monkeypatch.setenv("DD_PROFILING_API_TIMEOUT", "0.1")
+    stdout, stderr, exitcode, pid = call_program("python", os.path.join(os.path.dirname(__file__), "gevent_fork.py"))
+    assert exitcode == 0
diff --git a/tests/profiling/test_periodic.py b/tests/profiling/test_periodic.py
--- a/tests/profiling/test_periodic.py
+++ b/tests/profiling/test_periodic.py
@@ -54,14 +54,26 @@ def _on_shutdown():
     t.start()
     thread_started.wait()
     thread_continue.set()
+    assert t.is_alive()
     t.stop()
     t.join()
+    assert not t.is_alive()
     assert x["OK"]
     assert x["DOWN"]
     if hasattr(threading, "get_native_id"):
         assert t.native_id is not None
 
 
+def test_periodic_double_start():
+    def _run_periodic():
+        pass
+
+    t = _periodic.PeriodicRealThread(0.1, _run_periodic)
+    t.start()
+    with pytest.raises(RuntimeError):
+        t.start()
+
+
 def test_periodic_error():
     x = {"OK": False}
 
@@ -114,3 +126,11 @@ def test_periodic_join_stop_no_start():
     t.stop()
     t.join()
     t.stop()
+
+
+def test_is_alive_before_start():
+    def x():
+        pass
+
+    t = _periodic.PeriodicRealThread(1, x)
+    assert not t.is_alive()