Repository: dask/distributed

Issue:
Sporadic concurrency issue leading to zero task progress
Versions:
    
    - dask=0.14.3=py35_1
    - distributed=1.16.3=py35_0
    - tornado=4.5.1=py35_0

I'll try to create a repro but until I can do that, I'll just describe the setup here as a placeholder.  

I am running a LocalCluster with 6 processes (# cpus) and 1 thread per worker.

I am submitting tasks from tasks using `worker_client` and using an AsCompleted(loop=client.loop) in the worker_client context to await their completion. Occasionally I will see a secondary task complete (ie, the task-of-task has finished what it is supposed to do), but the original task (that launched it) continues to wait on its queue for notification of completion, halting progress of the program. When I interrupt the program, my stack traces are as follows:

In worker_client context / original task:

	^CTraceback (most recent call last):                                   
	  File "<string>", line 1, in <module>                                 
	  File "lib/python3.5/multiprocessing/forkserver.py", line 164, in main
		rfds = [key.fileobj for (key, events) in selector.select()]                                                                                                                     
	  File "lib/python3.5/selectors.py", line 441, in select
		fd_event_list = self._epoll.poll(timeout, max_ev)                                                           
	KeyboardInterrupt                                                      
	  File "...", line 248, in run
		for f in queue:                                                                                                 
	  File "lib/python3.5/site-packages/distributed/client.py", line 2634, in __next__
		return self.queue.get()                                                                         
	  File "lib/python3.5/queue.py", line 164, in get
		self.not_empty.wait()                                                                                    
	  File "lib/python3.5/threading.py", line 293, in wait
		waiter.acquire()    

Where the secondary task was running:

	tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f3fbd31b730>, <tornado.concurrent.Future object at 0x7f3fbd2c9d68>)
	Traceback (most recent call last):
	  File "lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
		ret = callback()
	  File "lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
		return fn(*args, **kwargs)
	  File "lib/python3.5/site-packages/tornado/ioloop.py", line 626, in _discard_future_result
		future.result()
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
		raise_exc_info(self._exc_info)
	  File "<string>", line 4, in raise_exc_info
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
		yielded = self.gen.throw(*exc_info)
	  File "lib/python3.5/site-packages/distributed/client.py", line 2600, in track_future
		yield _wait(future)
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
		value = future.result()
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
		raise_exc_info(self._exc_info)
	  File "<string>", line 4, in raise_exc_info
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1069, in run
		yielded = self.gen.send(value)
	  File "lib/python3.5/site-packages/distributed/client.py", line 2482, in _wait
		raise CancelledError(cancelled)
	concurrent.futures._base.CancelledError: ['mytask-349a99bf-9ffa-4fc6-988e-d85345a654d6']



Discussion:
EDIT: AsCompleted is a red herring. Locks up the same just client.gather(). 

This inevitably locks for me, whether after the first one or two iterations, or after several:

    """
    Repro of https://github.com/dask/distributed/issues/1176
    """

    from distributed import LocalCluster, worker_client
    from distributed.client import Client


    def task(x):
        return x


    def inner_runner(x):
        with worker_client() as client:
            q = []
            for i in range(1, x+1):
                f = client.submit(task, i, pure=False)
                q.append(f)

            tot = sum(client.gather(q))

            return tot


    def outer_runner(x):
        with worker_client() as client:
            q = []
            for i in range(1, x+1):
                f = client.submit(inner_runner, i, pure=False)
                q.append(f)

            tot = sum(client.gather(q))

            return tot


    def run_test():
        with LocalCluster() as cluster:
            with Client(cluster) as client:
                input('Run now')
                for _ in range(100):
                    futs = [client.submit(outer_runner, i, pure=False) for i in range(1, 11)]
                    # this will hang eventually
                    results = client.gather(futs)
                    print('Results = {}'.format(results))


    if __name__ == '__main__':
        run_test()
        input('done')


Output

    Results = [1, 4, 10, 20, 35, 56, 84, 120, 165, 220]
    Results = [1, 4, 10, 20, 35, 56, 84, 120, 165, 220]
    <hang after a few times>
Seems not to hang if `LocalCluster(n_workers=1)`, whereas `n_workers=2` gets it to hang for me.
Some more random info. If I set `inner_runner` to submit without `pure=False`, I still get hanging but with some scary log output (many of these per outermost iteration):

    distributed.scheduler - FATAL - Tried writing to closed comm: {'op': 'key-in-memory', 'key': 'task-644a381c188046862f8e700da0776916'}

First, thank you for going through the effort of the excellent reproducible example.  This is perfect.

Second, when I try this locally I'm actually unable to reproduce with either `pure=True` or `pure=False`.  Can I ask you to update to the newest version (just released a day ago) and see if the problem persists?
I just upgraded to the latest versions below, but unfortunately the problem persists. Do you also not get the FATAL log message if you toggle pure on the inner_runner?

    - dask=0.15.0=py35_0
    - distributed=1.17.0=py35_0
    - tornado=4.5.1=py35_0

(For me, sometimes it iterates as much as 30-35 times before locking.)

EDIT: I just tried this example at home on my (4 core) macbook air, and it does not lock up. Whereas at work I'm on a linux machine... 

    > cat /proc/version
    Linux version 4.1.35-pv-ts2 (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Mon Dec 5 21:15:47 GMT 2016

vs on macbook air

    > uname -r
    16.6.0
Possibly related:
https://bugs.python.org/issue25680

I'm seeing hangs here:

    rfds = [key.fileobj for (key, events) in selector.select()]
      File "lib/python3.5/selectors.py", line 441, in select
        fd_event_list = self._epoll.poll(timeout, max_ev)
    KeyboardInterrupt
I'm not sure whether this is related, but I hit this error in the same test that originally produced the deadlock I reported above:

	Traceback (most recent call last):                                                                                                                                                                        
	  File "mycode", line 34, in dask_client                                                                                                                  
		with worker_client(timeout=60) as client:                                                                                                                                                             
	  File "lib/python3.5/contextlib.py", line 59, in __enter__                                                                                                                
		return next(self.gen)                                                                                                                                                                                 
	  File "lib/python3.5/site-packages/distributed/worker_client.py", line 58, in worker_client                                                                               
		sync(wc.loop, gen.with_timeout, timedelta(seconds=timeout), wc._started)                                                                                                                              
	  File "lib/python3.5/site-packages/distributed/utils.py", line 234, in sync                                                                                               
		six.reraise(*error[0])                                                                                                                                                                                
	  File "lib/python3.5/site-packages/six.py", line 686, in reraise                                                                                                          
		raise value                                                                                                                                                                                           
	  File "lib/python3.5/site-packages/distributed/utils.py", line 223, in f                                                                                                  
		result[0] = yield make_coro()                                                                                                                                                                         
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                                                                     
		value = future.result()                                                                                                                                                                                                                                 
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                                                                                                                              
		raise_exc_info(self._exc_info)                                                                                                                                                                                                                          
	  File "<string>", line 4, in raise_exc_info                                                                                                                                                                                                                
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                                                                                                                       
		yielded = self.gen.throw(*exc_info)                                                                                                                                                                                                                     
	  File "lib/python3.5/site-packages/distributed/client.py", line 636, in _start                                                                                                                                                              
		yield self._ensure_connected(timeout=timeout)                                                                                                                                                                                                           
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                                                                                                                       
		value = future.result()                                                                                                                                                                                                                                 
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                                                                                                              
		raise_exc_info(self._exc_info)                                                                                                                                                                                                                          
	  File "<string>", line 4, in raise_exc_info                                                                                                                                                                                                                
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1069, in run                                                                                                                                                       
		yielded = self.gen.send(value)                                                                                                                                                                                                                          
	  File "lib/python3.5/site-packages/distributed/client.py", line 680, in _ensure_connected                                                                                                                                   
		assert len(msg) == 1                                                                                                                                                                                  
	AssertionError        

	During handling of the above exception, another exception occurred:                                                                
																														  
	Traceback (most recent call last):                                                                                    
	  File "lib/python3.5/site-packages/tornado/gen.py", line 307, in wrapper      
		yielded = next(result)                                                                                               
	  File "lib/python3.5/site-packages/distributed/client.py", line 832, in _shutdown     
		self._send_to_scheduler({'op': 'close-stream'})                                                           
	  File "lib/python3.5/site-packages/distributed/client.py", line 580, in _send_to_scheduler
		self.loop.add_callback(self.scheduler_comm.send, msg)                        
	AttributeError: 'NoneType' object has no attribute 'send'
	
	tornado.application - ERROR - Future <tornado.concurrent.Future object at 0x7fbfbf7aae48> exception was never retrieved: Traceback (most recent call last):                                               
        <... repeat of above Traceback ...>

Any idea what this might be?
Some encouraging news on this. Trying out https://github.com/dask/distributed/pull/1201 (which I've rebased on master to pick up latest changes as well), I edit `with worker_client() as client:` to instead be:

    client = get_client()
    # time.sleep(0.01)  # necessary?
    secede()

I do not get lockups anymore in this configuration.  If I don't stick in the `time.sleep`, I will occasionally get this and a resulting python exception (killedworker):

	tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7fc925c68d08>, 'inner_runner-7082c80f-4cd1-4279-aced-7a15f55f467a', 'long-running', compute_duration=1.3828277587890625e-05)
	Traceback (most recent call last):
	  File "/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
		ret = callback()
	  File "/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
		return fn(*args, **kwargs)
	  File "/home/aklein/src/distributed/distributed/worker.py", line 1287, in transition
		state = func(key, **kwargs)
	  File "/home/aklein/src/distributed/distributed/worker.py", line 1416, in transition_executing_long_running
		'compute_duration': compute_duration})
	  File "/home/aklein/src/distributed/distributed/batched.py", line 101, in send
		raise CommClosedError
	distributed.comm.core.CommClosedError

Also, if I leave `worker_client()` as in the original example, it doesn't proceed at all. If I stick a `time.sleep` after that line, I get this:

	tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f6727f5cea0>, <tornado.concurrent.Future object at 0x7f6727f5aa90>)
	Traceback (most recent call last):
	  File "/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
		ret = callback()
	  File "/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
		return fn(*args, **kwargs)
	  File "/lib/python3.5/site-packages/tornado/ioloop.py", line 626, in _discard_future_result
		future.result()
	  File "/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
		raise_exc_info(self._exc_info)
	  File "<string>", line 4, in raise_exc_info
	  File "/lib/python3.5/site-packages/tornado/gen.py", line 1069, in run
		yielded = self.gen.send(value)
	  File "/home/aklein/src/distributed/distributed/scheduler.py", line 3161, in reevaluate_occupancy
		self._reevaluate_occupancy_worker(w)
	  File "/home/aklein/src/distributed/distributed/scheduler.py", line 3181, in _reevaluate_occupancy_worker
		processing[key] = duration + comm
	TypeError: unsupported operand type(s) for +: 'NoneType' and 'float'


Can you print out or return the value of `get_client()._asynchronous` ?
also `get_client()._started.done()` and `get_client().status` if convenient
Actually in this case 

```
  File "/home/aklein/src/distributed/distributed/scheduler.py", line 3161, in reevaluate_occupancy
	self._reevaluate_occupancy_worker(w)
  File "/home/aklein/src/distributed/distributed/scheduler.py", line 3181, in _reevaluate_occupancy_worker
	processing[key] = duration + comm
```

Can you verify that you're using the most recent version of that branch?  I fixed an issue very similar to that recently.
I remain concerned about the need for a sleep though
Generally the approach in #1201 helps the problem here in two ways:

It avoids creating tons of clients, and instead reuses the same client in many threads.  To achieve this we made the client threadsafe (for common operations).  There is no reason to setup and teardown a client each time, you just use a client that is sitting around on the worker process.

It subtly tweaks scheduling of tasks submitted by tasks to try to prefer subtasks over super-tasks of the same generation.  This should stop a super-task-blowup in the common case.
Using `worker_client`, this test no longer locks up at all. Do you still want me to dig in to any of the above? I think this is solved. (As of commit 007cbf73f5992401cc11b0f847cfe942b0e64438)
No, I think we're good
Frustratingly, it seems that #1201 solves my reproduction of the issue, but "in the wild" I'm seeing things lock up. So it may be an entirely different root cause ... but the symptom looks the same. EDIT: I may have jumped the gun here, not sure I'm actually seeing lock ups. Need to stress test more today.
@adamklein your comment reached me by e-mail, but seems to have been deleted from github.  Was this intentional?  (it's also possible that github is being flaky)
I deleted it ... think I was seeing a red herring, thanks though!

On Thu, Jul 6, 2017 at 10:01 AM, Matthew Rocklin <notifications@github.com>
wrote:

> @adamklein <https://github.com/adamklein> your comment reached me by
> e-mail, but seems to have been deleted from github. Was this intentional?
> (it's also possible that github is being flaky)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1176#issuecomment-313404843>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA1DcNM9AvBvBTdUieSDTpwJ6Vw-83UQks5sLOivgaJpZM4N6agA>
> .
>

No longer a problem :)
Thought I had a repro... but was wrong. Still working on this.
I think I'm hitting this or something similar to this too. I'm doing interactive analysis in Jupyter on the with dask - distributed version `1.16.3`. Sometimes I go to do some computation and it just hangs. This is the stack trace I see. It's clear that no computation is taking place, and I can reproduce the issue with a `client.get()`.

Sometimes, nothing happens. Sometimes, there are processes listed in top consuming 100% CPU indefinitely (but they're not doing useful work).

<details>
<summary>Stack trace when interrupting `ddf.compute()`</summary>

```
KeyboardInterrupt                         Traceback (most recent call last)
<timed exec> in <module>()

~/.local/src/github.com/dask/dask/dask/base.py in compute(self, **kwargs)
     97             Extra keywords to forward to the scheduler ``get`` function.
     98         """
---> 99         (result,) = compute(self, traverse=False, **kwargs)
    100         return result
    101 

~/.local/src/github.com/dask/dask/dask/base.py in compute(*args, **kwargs)
    204     dsk = collections_to_dsk(variables, optimize_graph, **kwargs)
    205     keys = [var._keys() for var in variables]
--> 206     results = get(dsk, keys, **kwargs)
    207 
    208     results_iter = iter(results)

~/.local/lib/python3.6/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, **kwargs)
   1891         if sync:
   1892             try:
-> 1893                 results = self.gather(packed)
   1894             finally:
   1895                 for f in futures.values():

~/.local/lib/python3.6/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct)
   1353                 local_worker = None
   1354             return self.sync(self._gather, futures, errors=errors,
-> 1355                              direct=direct, local_worker=local_worker)
   1356 
   1357     @gen.coroutine

~/.local/lib/python3.6/site-packages/distributed/client.py in sync(self, func, *args, **kwargs)
    529             return future
    530         else:
--> 531             return sync(self.loop, func, *args, **kwargs)
    532 
    533     def __str__(self):

~/.local/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)
    230     loop.add_callback(f)
    231     while not e.is_set():
--> 232         e.wait(1000000)
    233     if error[0]:
    234         six.reraise(*error[0])

/usr/lib/python3.6/threading.py in wait(self, timeout)
    549             signaled = self._flag
    550             if not signaled:
--> 551                 signaled = self._cond.wait(timeout)
    552             return signaled
    553 

/usr/lib/python3.6/threading.py in wait(self, timeout)
    297             else:
    298                 if timeout > 0:
--> 299                     gotit = waiter.acquire(True, timeout)
    300                 else:
    301                     gotit = waiter.acquire(False)

KeyboardInterrupt:
```

</details>

&nbsp;
On the other side I now see two processes spinning using 100% CPU, and pyflame shows more or less the same stack trace as in #1313 - that it is in `threading.wait`. I'm now on python 3.6 rather than python 2.6.

Some other details about my current state:

I started this local distributed cluster in my jupyter notebook 12 hours ago, came back after leaving tensorflow training overnight, and then tried running some dask operations, and they don't work.

I tried doing `client.restart()`, which does return but appears to have no effect. I also tried sending `kill -INT` to the pids which are spinning using CPU, to no avail.

<details>

<summary>perf top on one of the processes which is spinning using CPU</summary>

```
  19.19%  python3.6                              [.] _PyEval_EvalFrameDefault                                                                                                    
   4.70%  python3.6                              [.] _PyType_Lookup                                                                                                              
   4.28%  python3.6                              [.] PyObject_GenericGetAttr                                                                                                     
   2.49%  python3.6                              [.] PyObject_GetAttr                                                                                                            
   2.37%  python3.6                              [.] PyDict_GetItem                                                                                                              
   2.12%  [kernel]                               [k] __lock_text_start                                                                                                           
   1.93%  _poll.cpython-36m-x86_64-linux-gnu.so  [.] __pyx_pf_3zmq_7backend_6cython_5_poll_zmq_poll.isra.15                                                                      
   1.91%  python3.6                              [.] PyFrame_New                                                                                                                 
   1.81%  python3.6                              [.] _PyDict_LoadGlobal                                                                                                          
   1.45%  [kernel]                               [k] tcp_poll                                                                                                                    
   1.38%  [kernel]                               [k] __fget                                                                                                                      
   1.27%  python3.6                              [.] 0x00000000000f4ceb                                                                                                          
   1.12%  python3.6                              [.] PyType_IsSubtype                                                                                                            
   0.97%  python3.6                              [.] PyTuple_New                                                                                                                 
   0.95%  python3.6                              [.] _PyCFunction_FastCallKeywords                                                                                               
   0.90%  [kernel]                               [k] sock_poll                                                                                                                   
   0.84%  [kernel]                               [k] do_sys_poll      
   0.80%  python3.6                              [.] _PyDict_GetItemId
   0.76%  [kernel]                               [k] fput                                                                                                                        
   0.75%  python3.6                              [.] PyUnicode_FromFormatV                                                                                                       
   0.63%  python3.6                              [.] 0x000000000006933b                                                                                                          
   0.62%  [kernel]                               [k] entry_SYSCALL_64_fastpath
   0.61%  python3.6                              [.] PyDict_New                                                                                                                  
   0.58%  python3.6                              [.] PyObject_GC_UnTrack                                                                                                         
   0.56%  python3.6                              [.] _Py_CheckFunctionResult
   0.55%  python3.6                              [.] 0x00000000001988ae     
   0.55%  python3.6                              [.] _PyObject_FastCallDict                                                                                                      
   0.50%  python3.6                              [.] 0x00000000000669ff     
   0.46%  python3.6                              [.] _PyObjectDict_SetItem
   0.45%  libc-2.24.so                           [.] malloc
   0.44%  libzmq-76f20544.so.5.0.1               [.] zmq_poll                
   0.44%  python3.6                              [.] PyArg_UnpackTuple       
   0.44%  [kernel]                               [k] pvclock_clocksource_read                                                                                                    
   0.40%  python3.6                              [.] PyObject_RichCompareBool
   0.40%  python3.6                              [.] PyObject_GetItem                                                                                                            
   0.39%  python3.6                              [.] PyErr_Occurred
   0.39%  python3.6                              [.] PyList_New
   0.38%  python3.6                              [.] PyMethod_New              
   0.38%  python3.6                              [.] 0x00000000000f4be0        
   0.37%  python3.6                              [.] PyFunction_NewWithQualName                                                                                                  
   0.34%  python3.6                              [.] 0x000000000006934e                                                                                                          
   0.34%  python3.6                              [.] PyObject_SetAttr                                                                                                            
   0.30%  python3.6                              [.] 0x00000000000ce3c0         
   0.29%  python3.6                              [.] 0x00000000000f4758                                                                                                          
   0.29%  libpthread-2.24.so                     [.] pthread_mutex_lock
   0.28%  python3.6                              [.] _PyThreadState_UncheckedGet
   0.26%  python3.6                              [.] PyObject_IsInstance                                                                                                         
   0.26%  python3.6                              [.] 0x000000000019863d   
   0.25%  [vdso]                                 [.] __vdso_clock_gettime
   0.25%  python3.6                              [.] PyDict_SetItem      
   0.24%  python3.6                              [.] PyFrame_BlockSetup                                                                                                          
   0.23%  python3.6                              [.] 0x0000000000198629                                                                                                          
   0.23%  [kernel]                               [k] tcp_stream_memory_free                                                                                                      
   0.23%  python3.6                              [.] PyUnicode_RichCompare                                                                                                       
   0.23%  python3.6                              [.] PyObject_Call         
   0.22%  python3.6                              [.] 0x00000000000ce42c                                                                                                          
   0.22%  python3.6                              [.] 0x00000000001986f6
   0.21%  libc-2.24.so                           [.] strlen                                                                                                                      
   0.21%  libpthread-2.24.so                     [.] pthread_cond_signal@@GLIBC_2.3.2                                                                                            
   0.21%  python3.6                              [.] PyLong_FromLong                                                                                                             
   0.20%  python3.6                              [.] _PyObject_GC_New    
   0.19%  python3.6                              [.] PyObject_RichCompare            
   0.18%  python3.6                              [.] 0x00000000000f4cc0                                                                                                          
   0.17%  python3.6                              [.] _PyObject_GenericGetAttrWithDict
   0.17%  python3.6                              [.] PyObject_GetIter                                                                                                            
   0.17%  python3.6                              [.] 0x00000000001982e6
   0.17%  python3.6                              [.] 0x000000000011c004          
```

</details>

&nbsp;
Here are the top few python stack traces (number of samples on the left, stacks on the right).

<details>

<summary>Python stacks</summary>

```
8344 - /usr/lib/python3.6/threading.py:_bootstrap:871;/usr/lib/python3.6/threading.py:_bootstrap_inner:917;/usr/lib/python3.6/threading.py:run:853;/usr/lib/python3.6/concurrent/futures/thread.py:_worker:66;/usr/lib/python3.6/queue.py:get:153;/usr/lib/python3.6/threading.py:wait:274
8335 - /usr/lib/python3.6/threading.py:_bootstrap:871;/usr/lib/python3.6/threading.py:_bootstrap_inner:917;/usr/lib/python3.6/threading.py:run:853;/home/pwaller/.local/lib/python3.6/site-packages/distributed/process.py:_watch:120;/home/pwaller/.local/lib/python3.6/site-packages/distributed/process.py:_process_one_message:122;/usr/lib/python3.6/queue.py:get:230;/usr/lib/python3.6/threading.py:wait:278
2086 - /usr/lib/python3.6/threading.py:_bootstrap:871;/usr/lib/python3.6/threading.py:_bootstrap_inner:917;/usr/lib/python3.6/threading.py:run:853;/home/pwaller/.local/lib/python3.6/site-packages/ipykernel/iostream.py:_thread_main:70;/home/pwaller/.local/lib/python3.6/site-packages/zmq/eventloop/ioloop.py:start:176;/home/pwaller/.local/lib/python3.6/site-packages/tornado/ioloop.py:start:865;/home/pwaller/.local/lib/python3.6/site-packages/zmq/eventloop/ioloop.py:poll:118;/home/pwaller/.local/lib/python3.6/site-packages/zmq/sugar/poll.py:poll:77
2086 - /usr/lib/python3.6/threading.py:_bootstrap:871;/usr/lib/python3.6/threading.py:_bootstrap_inner:917;/home/pwaller/.local/lib/python3.6/site-packages/zmq/utils/garbage.py:run:44
2086 - /usr/lib/python3.6/threading.py:_bootstrap:871;/usr/lib/python3.6/threading.py:_bootstrap_inner:917;/home/pwaller/.local/lib/python3.6/site-packages/ipykernel/parentpoller.py:run:34
2086 - /usr/lib/python3.6/threading.py:_bootstrap:871;/usr/lib/python3.6/threading.py:_bootstrap_inner:917;/home/pwaller/.local/lib/python3.6/site-packages/ipykernel/heartbeat.py:run:174
2086 - /usr/lib/python3.6/threading.py:_bootstrap:871;/usr/lib/python3.6/threading.py:_bootstrap_inner:917;<decorator-gen-24>:run:2;/home/pwaller/.local/lib/python3.6/site-packages/IPython/core/history.py:needs_sqlite:52;/home/pwaller/.local/lib/python3.6/site-packages/IPython/core/history.py:run:823;/usr/lib/python3.6/threading.py:wait:534;/usr/lib/python3.6/threading.py:wait:274
2086 - /usr/lib/python3.6/runpy.py:_run_module_as_main:171;/usr/lib/python3.6/runpy.py:_run_code:74;/home/pwaller/.local/lib/python3.6/site-packages/ipykernel_launcher.py:<module>:6;/home/pwaller/.local/lib/python3.6/site-packages/traitlets/config/application.py:launch_instance:651;/home/pwaller/.local/lib/python3.6/site-packages/ipykernel/kernelapp.py:start:471;/home/pwaller/.local/lib/python3.6/site-packages/zmq/eventloop/ioloop.py:start:176;/home/pwaller/.local/lib/python3.6/site-packages/tornado/ioloop.py:start:865;/home/pwaller/.local/lib/python3.6/site-packages/zmq/eventloop/ioloop.py:poll:118;/home/pwaller/.local/lib/python3.6/site-packages/zmq/sugar/poll.py:poll:77
```

</details>
@pwaller In my case, no CPU is being burned; all threads are sitting idle on their queues and waiting for work and/or results (in the case of the client that should be proceeding but is not). The reason the worker client cannot proceed is because the result of a submitted task is never returned (ie the client is never notified), even though that submitted task is run successfully through to completion.

I'm deep into tracing this with timestamped line-debugging, and I've narrowed it down to this:

On the worker-client side that never gets a result, it submits a task that goes through `client.submit` successfully. It enters its `_graph_to_futures` and constructs its message and queues it up to send to the scheduler here:
https://github.com/dask/distributed/blob/master/distributed/client.py#L1861

The scheduler, however, never processes this message correctly, and therefore does not update its `who_wants` state via `client_desires_keys` here:
https://github.com/dask/distributed/blob/master/distributed/scheduler.py#L670

This task is always one that has been submitted by another worker-client as well, usually within microseconds.  Because the second worker-client is not in the `who_wants` dictionary, when the task finishes, said worker-client is never notified that its task submission is completed.

I'm currently trying to figure out if the scheduler receives a garbled message and drops it, or if it never receives the communication, or what...

There are other interesting things about the scheduler state that makes this a corner case. The task that is submitted by the worker client is always already running, having been submitted by another worker (usually within microseconds); I also think the task has usually been stolen and/or released in its transition history.

Anyway, I feel that I keep getting closer to the answer...


Sorry for the slow response on this.  I'm working through a backlog of
scheduler issues after being away for a bit on vacation.

On Fri, Aug 11, 2017 at 11:09 AM, Adam Klein <notifications@github.com>
wrote:

> @pwaller <https://github.com/pwaller> In my case, no CPU is being burned;
> all threads are sitting idle on their queues and waiting for work and/or
> results (in the case of the client that should be proceeding but is not).
> The reason the cleint cannot proceed is because the result of a submitted
> task is never submitted back, even though that submitted task is run
> successfully through to completion.
>
> I'm deep into tracing this with timestamped line-debugging, and I've
> narrowed it down to this:
>
> On the worker-client side, it submits a task that goes through
> client.submit successfully. It enters its _graph_to_futures and
> constructs its message and queues it up to send to the scheduler here:
> https://github.com/dask/distributed/blob/master/
> distributed/client.py#L1861
>
> The scheduler, however, never processes this message correctly, and
> therefore does not update its who_wants state via client_desires_keys
> here:
> https://github.com/dask/distributed/blob/master/
> distributed/scheduler.py#L670
>
> That means when the task finishes, the worker-client that fired off the
> task never is notified that its task submission is completed.
>
> I'm currently trying to figure out if the scheduler receives a garbled
> message and drops it, or if it never receives the communication, or what...
>
> There are other interesting things about the state that makes this a
> corner case. The task that is submitted by the worker client is always
> already running, having been submitted by another worker (usually within
> microseconds); I also think the task has usually been stolen and/or
> released in its history.
>
> Anyway, I feel that I keep getting closer to the answer...
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1176#issuecomment-321839608>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszBY7v_rAJhlLAJzVd8txvzSPzEhRks5sXG6_gaJpZM4N6agA>
> .
>

Ok, I've isolated where the message is lost.  Starting here:

https://github.com/dask/distributed/blob/master/distributed/client.py#L634

The message that is getting lost is scheduled, via a callback, to be buffered into client's `_pending_msg_buffer`, as the client status is `connecting`. The `_pending_msg_buffer` is supposed to be emptied and its messages transmitted in an ensuing call to `_ensure_connected`. However, this line is never subsequently hit:

https://github.com/dask/distributed/blob/master/distributed/client.py#L744

I haven't analyzed where the race condition is happening yet, but this is the culprit. (E.g., could it be that `_ensure_connected` is invoked before the `_pending_msg_buffer.append` callback?)

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -627,11 +627,18 @@ def start(self, **kwargs):
     def __await__(self):
         return self._started.__await__()
 
-    def _send_to_scheduler(self, msg):
+    def _send_to_scheduler_safe(self, msg):
         if self.status in ('running', 'closing'):
-            self.loop.add_callback(self.scheduler_comm.send, msg)
+            self.scheduler_comm.send(msg)
         elif self.status is 'connecting':
-            self.loop.add_callback(self._pending_msg_buffer.append, msg)
+            self._pending_msg_buffer.append(msg)
+        else:
+            raise Exception("Tried sending message after closing.  Status: %s\n"
+                            "Message: %s" % (self.status, msg))
+
+    def _send_to_scheduler(self, msg):
+        if self.status in ('running', 'closing', 'connecting'):
+            self.loop.add_callback(self._send_to_scheduler_safe, msg)
         else:
             raise Exception("Tried sending message after closing.  Status: %s\n"
                             "Message: %s" % (self.status, msg))


Test patch: