Repository: dask/distributed

Issue:
Semaphores release automatically - refresh won't run
**What happened**:
Semaphores always automatically release after 40 seconds

```
Example output:
1597944899.78569 1
1597944899.7857327 0
1597944939.7501612 3
1597944939.75026 2
distributed.semaphore - WARNING - Tried to release semaphore but it was already released: name=database, lease_id=52a2ef49dbfa4fafb68d1118adfcb807. This can happen if the semaphore timed out before.
distributed.semaphore - WARNING - Tried to release semaphore but it was already released: name=database, lease_id=08821fab060c4d25b05209d5bfba6716. This can happen if the semaphore timed out before.
finished 0
finished 1
...
```

**What you expected to happen**:
Semaphore leases should be automatically refreshed.

**Minimal Complete Verifiable Example**:

```python
import dask.distributed as dd

import time


if __name__ == '__main__':

    client = dd.Client(processes=False, threads_per_worker=10)

    sem = dd.Semaphore(max_leases=2, name="database")

    def access_limited(val, sem):
        with sem:
            print(time.time(), val)
            # Interact with the DB
            time.sleep(60)
        print('finished', val)

    futures = client.map(access_limited, range(10), sem=sem)
    client.gather(futures)
    sem.close()
```

**Anything else we need to know?**:
Workaround would be appreciated!

**Environment**:

- Dask version: 2.22.0 & 2.23.0 tested
- Python version: 3.7
- Operating System: Linux
- Install method (conda, pip, source): pip



Discussion:
The example is exactly a copy from the docs (adjusted to add the sleep and create a client). I also tried to make access_limited a asyncio coroutine, but that won't actually run.
Thanks for raising an issue @pvanderlinden 

cc'ing @fjetter and @lr4d who I think did most of the work here and may have thoughts.
Also more remarks: during a task, the ioloop is not running, which is the reason. Do I need to managed the ioloop myself if I want to use a semaphore in a task (just trying to figure out how they currently can be used for a workaround)?
The behaviour you're describing is an expected state if you're executing a computation which holds the GIL but this is clearly not the case with the example (I can confirm the example on my end)

> during a task, the ioloop is not running, which is the reason. Do I need to managed the ioloop myself if I want to use a semaphore in a task (just trying to figure out how they currently can be used for a workaround)?

We do require an event loop to be active but there should not be a need to manage it yourself. this should be done by the client. I'll need to dig a bit deeper to determine the root cause. In the meantime, increasing the lease timeout should serve as a workaround to ensure the leases simply never time out (that's a protection for cases where workers die and cannot release the semaphore again). That's the configuration option `distributed.scheduler.locks.lease-timeout` which is, by default, set to 30s.

The issue appears to be that the client-side refresh `PeriodicCallback` is never actually executed (it is scheduled but never executed)

https://github.com/dask/distributed/blob/6a32208717c04aa9ccb4defdf253b8a00cab32b7/distributed/semaphore.py#L367-L382

My suspicion is that in this example we're running workers in a threadpool. According to https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading "Almost all asyncio objects are not thread safe" and I assume this is then also true for the tornado PC and this is why the PC is never properly scheduled? If that's the case, I'd be surprised to see this only pop up now, though. @mrocklin has something like this ever occurred / is this a known issue?
> The behaviour you're describing is an expected state if you're executing a computation which holds the GIL but this is clearly not the case with the example (I can confirm the example on my end)
> 
> > during a task, the ioloop is not running, which is the reason. Do I need to managed the ioloop myself if I want to use a semaphore in a task (just trying to figure out how they currently can be used for a workaround)?
> 
> We do require an event loop to be active but there should not be a need to manage it yourself. this should be done by the client. I'll need to dig a bit deeper to determine the root cause. In the meantime, increasing the lease timeout should serve as a workaround to ensure the leases simply never time out (that's a protection for cases where workers die and cannot release the semaphore again). That's the configuration option `distributed.scheduler.locks.lease-timeout` which is, by default, set to 30s.

Thanks, that needs to be a high timeout though (30min or so, using it to parallelize complex larger python tasks).

Not sure if I should ask this in this issue, but I also do something from preventing the tasks to be waiting for a sem and using up executors. I call `secede()` before trying to acquire the lease, and `rejoin()` after the I acquired the lease. Can that cause issues? It's not causing issues with the refresh callback though, as the simple example has the issue as well.
@fjetter You are correct, and not correct I think. I tested a few two things:
- Using ioloop.add_callback, which is threadsafe: doesn't fix the issue
- Not using threads: `client = dd.Client(processes=False, threads_per_worker=10)` -> `client = dd.Client(processes=10, threads_per_worker=1)` Which fixes the the issue (I will use this workaround, as it might be better to not use threads for my workload anyway). 

Not entirely sure (not 100% aware of Tornado's and Dask internals), but: PeriodicCallback uses `IOLoop.current()`, which might grab the IOLoop from a worker running in a different thread?

Resolving patch:
diff --git a/distributed/semaphore.py b/distributed/semaphore.py
--- a/distributed/semaphore.py
+++ b/distributed/semaphore.py
@@ -362,6 +362,12 @@ def __init__(self, max_leases=1, name=None, client=None, register=True):
         self.id = uuid.uuid4().hex
         self._leases = deque()
 
+        self.refresh_leases = True
+
+        self._registered = None
+        if register:
+            self._registered = self.register()
+
         # this should give ample time to refresh without introducing another
         # config parameter since this *must* be smaller than the timeout anyhow
         refresh_leases_interval = (
@@ -371,7 +377,6 @@ def __init__(self, max_leases=1, name=None, client=None, register=True):
             )
             / 5
         )
-        self._refreshing_leases = False
         pc = PeriodicCallback(
             self._refresh_leases, callback_time=refresh_leases_interval * 1000
         )
@@ -379,12 +384,10 @@ def __init__(self, max_leases=1, name=None, client=None, register=True):
         # Registering the pc to the client here is important for proper cleanup
         self._periodic_callback_name = f"refresh_semaphores_{self.id}"
         self.client._periodic_callbacks[self._periodic_callback_name] = pc
-        pc.start()
-        self.refresh_leases = True
 
-        self._registered = None
-        if register:
-            self._registered = self.register()
+        # Need to start the callback using IOLoop.add_callback to ensure that the
+        # PC uses the correct event lopp.
+        self.client.io_loop.add_callback(pc.start)
 
     def register(self):
         """


Test patch:
diff --git a/distributed/tests/test_semaphore.py b/distributed/tests/test_semaphore.py
--- a/distributed/tests/test_semaphore.py
+++ b/distributed/tests/test_semaphore.py
@@ -11,6 +11,7 @@
 from distributed.metrics import time
 from distributed.utils_test import (  # noqa: F401
     client,
+    cleanup,
     cluster,
     async_wait_for,
     captured_logger,
@@ -491,3 +492,29 @@ async def test_metrics(c, s, a, b):
         "pending": defaultdict(int, {"test": 0}),
     }
     assert actual == expected
+
+
+def test_threadpoolworkers_pick_correct_ioloop(cleanup):
+    # gh4057
+
+    with dask.config.set(
+        {
+            "distributed.scheduler.locks.lease-validation-interval": 0.01,
+            "distributed.scheduler.locks.lease-timeout": 0.05,
+        }
+    ):
+        with Client(processes=False, threads_per_worker=4) as client:
+            sem = Semaphore(max_leases=1, name="database")
+            protected_ressource = []
+
+            def access_limited(val, sem):
+                import time
+
+                with sem:
+                    assert len(protected_ressource) == 0
+                    protected_ressource.append(val)
+                    # Interact with the DB
+                    time.sleep(0.1)
+                    protected_ressource.remove(val)
+
+            client.gather(client.map(access_limited, range(10), sem=sem))