Repository: dask/distributed

Issue:
distributed.worker - ERROR - list index out of range
```
# environment
dask.__version__ : '1.2.2'
distributed.__version__:  '1.28.0'
```
We got the following error from distributed.worker

```
# error message
distributed.worker - ERROR - list index out of range
Traceback (most recent call last):
  File "/usr/local/anaconda3/lib/python3.6/site-packages/distributed/worker.py", line 2336, in execute
    self.transition(key, "memory", value=value)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/distributed/worker.py", line 1443, in transition
    state = func(key, **kwargs)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/distributed/worker.py", line 1562, in transition_executing_done
    self.send_task_state_to_scheduler(key)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/distributed/worker.py", line 1727, in send_task_state_to_scheduler
    typ_serialized = dumps_function(typ)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/distributed/worker.py", line 3040, in dumps_function
    result = cache[func]
  File "/usr/local/anaconda3/lib/python3.6/site-packages/zict/lru.py", line 50, in __getitem__
    self.heap[key] = self.i
  File "/usr/local/anaconda3/lib/python3.6/site-packages/heapdict.py", line 39, in __setitem__
    self.pop(key)
  File "/usr/local/anaconda3/lib/python3.6/_collections_abc.py", line 801, in pop
    del self[key]
  File "/usr/local/anaconda3/lib/python3.6/site-packages/heapdict.py", line 78, in __delitem__
    self._swap(wrapper[2], parent[2])
  File "/usr/local/anaconda3/lib/python3.6/site-packages/heapdict.py", line 68, in _swap
    self.heap[i], self.heap[j] = self.heap[j], self.heap[i]
IndexError: list index out of range
/usr/local/anaconda3/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 48 leaked semaphores to clean up at shutdown
  len(cache))
```

We trace the issue, and we figure out this might be caused by the change in recent update in distributed.worker
```python
# copy from distributed/worker.py
try:
    # a 10 MB cache of deserialized functions and their bytes
    from zict import LRU
    cache = LRU(10000000, dict(), weight=lambda k, v: len(v))
except ImportError:
    cache = dict()

def dumps_function(func):
    """ Dump a function to bytes, cache functions """
    try:
        result = cache[func]
    except KeyError:
        result = pickle.dumps(func)
        if len(result) < 100000:
            cache[func] = result
    except TypeError:
        result = pickle.dumps(func)
    return result
```
 In recent change, distributed.worker use zict LRU as cache, but the LRU's __get_item__ is not thread safe. The following is the minimal example to reproduce the " list index out of range" error purely use LRU.
```python
from zict import LRU
from functools import partial
import concurrent.futures
# create LRU cache
cache=LRU(2,dict())
cache[1]=1
cache[2]=2
# function to get key from cache multiple times
def get_key(key,reps):
    for _ in range(reps):
        cache[key]

get_key_m=partial(get_key,reps=1000000)
# test get key from multiple threads 
def test_get_key():
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        [i for i in executor.map(get_key_m, [1,2])]
    
# this call will provide "IndexError: list index out of range"
test_get_key()

```

We are not able to figure out a minimal example to produce the list index out of range by using Dask directly, and hopefully the minimal example of using LRU can provide some insights. Please let me know your opinion. Thanks!


Discussion:
Thanks for this. I'm going to spend a bit of time trying to get an example working with distributed.

FWIW, I ran your minimal example, but didn't get a failure after ~10 minutes. Do you know how long you ran it for? What version of zict?
For the record, this does not produce an IndexError for me

```python
import numpy as np
import dask
from dask import delayed
from distributed import Client, LocalCluster


def f(x):
    return np.random.random(int(x))


def g(*args):
    return 0


def main():
    cluster = LocalCluster(memory_limit="1000Mb", n_workers=4, threads_per_worker=8)

    with Client(cluster):
        input_sizes = [100] * 1000
        inputs = [delayed(f)(i) for i in input_sizes]
        output = [delayed(g)(*inputs) for _ in range(10)]

        print('computing')
        dask.compute(*output)
        print('done')

    cluster.close()


if __name__ == '__main__':
    main()
```


I've confirmed that we're spilling to disk. Beyond that, I'm not sure if I'm on the right path, and just need to tweak things like the size / timing of the tasks / number of threads, or whether this is completely off base.
Thanks for the quick response.
the zict version I use is '0.1.3'
For the minimal example that I provide, I do get stuck when I run the example sometimes(normally I will restart the kernel and run it again)... but if the run fails with "list index out of range", it will happen very quickly. 
I tested the code snippet I provided in pycharm's python console.

I am seeing this multiprocessing semaphore error, too. I do not have a reproducible example, but I can report that I am using distributed 1.28.1 and zict 0.1.4.
Our set up does not spill to disk.
Given the lack of a reproducible example, would either of you be willing to try placing a lock around access to the cache object and see if that solves your problem?  This would probably be something like the following:

```diff
diff --git a/distributed/worker.py b/distributed/worker.py
index 667bd834..da3ec83e 100644
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -3097,14 +3097,19 @@ except ImportError:
     cache = dict()


+_cache_lock = threading.Lock()
+
+
 def dumps_function(func):
     """ Dump a function to bytes, cache functions """
     try:
-        result = cache[func]
+        with _cache_lock:
+            result = cache[func]
     except KeyError:
         result = pickle.dumps(func)
         if len(result) < 100000:
-            cache[func] = result
+            with _cache_lock:
+                cache[func] = result
     except TypeError:
         result = pickle.dumps(func)
     return result
```
@quickpanda @caseyjlaw are either of you able to try out https://github.com/dask/distributed/issues/2727#issuecomment-495922933?
@TomAugspurger, by adding the lock into the mini-example I provided, it resolved list index out of range error.
I will try to modify the dask code to see if it resolve the issue in our application
@TomAugspurger @mrocklin, I just tried the lock modification on distributed source code and run with our application, and "list index out of range" error got resolved! Thanks for the input, and hopefully this fix can be in the recent release.
@TomAugspurger
We recently also see another distributed worker error related to heapq.heappush and heapq.heappop operation inside the worker. Since there errors didn't always appear,  I am not able to come out a concrete example. I am not sure if this is also related to threads issues. 

Here are some error message:
```
distributed.worker - ERROR - '<' not supported between instances of 'NoneType' and 'tuple'
Traceback (most recent call last):
  File "/home/binyu/.conda/envs/forecast-env/lib/python3.6/site-packages/distributed/worker.py", line 1468, in transition_waiting_ready
    heapq.heappush(self.ready, (self.priorities[key], key))
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
distributed.worker - ERROR - '<' not supported between instances of 'NoneType' and 'tuple'
Traceback (most recent call last):
  File "/home/binyu/.conda/envs/forecast-env/lib/python3.6/site-packages/distributed/worker.py", line 1317, in add_task
    self.transition(key, "ready")
  File "/home/binyu/.conda/envs/forecast-env/lib/python3.6/site-packages/distributed/worker.py", line 1443, in transition
    state = func(key, **kwargs)
  File "/home/binyu/.conda/envs/forecast-env/lib/python3.6/site-packages/distributed/worker.py", line 1468, in transition_waiting_ready
    heapq.heappush(self.ready, (self.priorities[key], key))
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
distributed.core - ERROR - '<' not supported between instances of 'NoneType' and 'tuple'
```
```
distributed.worker - ERROR - '<' not supported between instances of 'NoneType' and 'tuple'
Traceback (most recent call last):
  File "/home/binyu/.conda/envs/forecast-env/lib/python3.6/site-packages/distributed/worker.py", line 2261, in ensure_computing
    _, key = heapq.heappop(self.ready)
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
distributed.worker - ERROR - '<' not supported between instances of 'NoneType' and 'tuple'
Traceback (most recent call last):
  File "/home/binyu/.conda/envs/forecast-env/lib/python3.6/site-packages/distributed/worker.py", line 2366, in execute
    self.ensure_computing()
  File "/home/binyu/.conda/envs/forecast-env/lib/python3.6/site-packages/distributed/worker.py", line 2261, in ensure_computing
    _, key = heapq.heappop(self.ready)
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f5abba97748>>, <Future finished exception=TypeError("'<' not supported between instances of 'NoneType' and 'tuple'",)>)
```

Thanks, I'm poking around this area at the moment. Is your application using user-defined priorities when submitting tasks?
@TomAugspurger We are not using user-defined priorities to submit tasks, but we will submit tasks when on the worker nodes.
@TomAugspurger for the heapq.heappush issue, we noticed that when the value corresponding to the key in priorities is None, it will cause  
```
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
```
For example, the following code snippet will get the error message
```python
import heapq
q=[]
heapq.heappush(q,(None,1))
heapq.heappush(q,(None,2))
heapq.heappush(q,((1,),3))
```
In our application, we are not using priorities to submit task, but will launch task from worker, and when we are testing small data set, the worker seems fine. However when we test with large dataset, the issue will come out.


Thanks. I haven't had a chance to look into either of these more closely yet.

@quickpanda one thing that would be helpful, can you add a bit of debugging code to your worker to print out the priorities? Either set `distributed.admin.pdb-on-err` or something like 

```diff
diff --git a/distributed/worker.py b/distributed/worker.py
index 37dcbc2e..35d8a2bd 100644
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -1472,6 +1472,7 @@ class Worker(ServerNode):
                 heapq.heappush(self.ready, (self.priorities[key], key))
         except Exception as e:
             logger.exception(e)
+            logger.warning('Bad priority %s', self.priorities['key'])
             if LOG_PDB:
                 import pdb
 
```

it may be helpful to know which element of the tuple is None.
Hi, guys. 

Running into  ```File "/usr/local/anaconda3/lib/python3.6/site-packages/heapdict.py", line 68, in _swap
    self.heap[i], self.heap[j] = self.heap[j], self.heap[i]
IndexError: list index out of range```
when calling `client.compute(delayed(task))` inside `threading.Thread`


e.g 

```
def task():
    # some stuff here

def consume():
    while True:
        client.compute(delayed(task))

consumer = threading.Thread(target=consume)
consumer.start()
```

```
dask                 2.1.0     
dask-glm             0.1.0     
dask-ml              1.0.0     
```

inserting ```_cache_lock = threading.Lock()``` seems to solve the problem. 

Thanks @Techn0logic .  Are you able to make a minimal failing example, perhaps something that we could use as a test going forward?  This is really the hardest part of this process, and it looks like you might already be part of the way there.
We were having a problem with occasional deadlocks on distributed cluster, eventually I narrowed it down to the traceback in the OP. Placing a thread lock on cache's `__getitem__` and `__setitem__` like in [#2727 (comment)](https://github.com/dask/distributed/issues/2727#issuecomment-495922933) appears to eliminate the problem.

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -3234,15 +3234,19 @@ def execute_task(task):
 
 cache_dumps = LRU(maxsize=100)
 
+_cache_lock = threading.Lock()
+
 
 def dumps_function(func):
     """ Dump a function to bytes, cache functions """
     try:
-        result = cache_dumps[func]
+        with _cache_lock:
+            result = cache_dumps[func]
     except KeyError:
         result = pickle.dumps(func)
         if len(result) < 100000:
-            cache_dumps[func] = result
+            with _cache_lock:
+                cache_dumps[func] = result
     except TypeError:  # Unhashable function
         result = pickle.dumps(func)
     return result


Test patch: