Repository: dask/distributed

Issue:
ValueError / hang on LocalCluster startup
I'm loading my conda environment off NAS storage, and when usage is heavy, there can be long delays in loading things, which I believe can exceed the default timeout (five seconds) here:

https://github.com/dask/distributed/blob/master/distributed/client.py#L428

As a result I see this on a call to `LocalCluster()`:

	Traceback (most recent call last):                                                                                                                            
	  File "lib/python3.5/site-packages/tornado/gen.py", line 910, in error_callback                                                
		future.result()                                                                                                                                           
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)                                                                                                                            
	  File "<string>", line 4, in raise_exc_info                                                                                                                  
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "lib/python3.5/site-packages/distributed/nanny.py", line 300, in start
		yield self._wait_until_running()                                           
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()                                                                                                                                   
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
		raise_exc_info(self._exc_info)                                             
	  File "<string>", line 4, in raise_exc_info                                                                                                                  
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1069, in run
		yielded = self.gen.send(value)                                                                                                                            
	  File "lib/python3.5/site-packages/distributed/nanny.py", line 386, in _wait_until_running
		raise ValueError("Worker not started")                                                                                                                                                                                                                                                                                   
	ValueError: Worker not started         

This `ValueError` does not seem to be passed up to my application; my application seems frozen waiting for the cluster start up (not 100% sure where) - ie, it never returns from the `LocalCluster()` call. Although the multiprocessing pool actually does start up, in this case too late to matter.

How can I catch / retry this situation?


Discussion:
Or, it could have nothing to do with a timeout. :) I see the guard has to do with the status, and there's an iffy looking `delay = 0.05` on line 383 of nanny.py ...

EDIT:  hmm, I'll try to see what the `self.status` is instead of `starting`, I guess that's the key here ...
Ok, I seem to have induced a ValueError with a indistriminately sent SIGINT (Ctrl-C) which shows up as the following:

	 File "lib/python3.5/multiprocessing/forkserver.py", line 164, in main        
		rfds = [key.fileobj for (key, events) in selector.select()]                                                                                               
	  File "lib/python3.5/selectors.py", line 445, in select
		fd_event_list = self._epoll.poll(timeout, max_ev)                                                                                                         
	KeyboardInterrupt                                                                             

	tornado.application - ERROR - Multiple exceptions in yield list                
	Traceback (most recent call last):     
	  File "lib/python3.5/site-packages/tornado/gen.py", line 828, in callback                                                      
		result_list.append(f.result())     
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/deploy/local.py", line 173, in _start_worker                                                                 
		yield w._start()                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 135, in _start                                                                               
		response = yield self.instantiate()                                        
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 189, in instantiate                                                                          
		self.process.start())              
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 300, in start 
		yield self._wait_until_running()   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1069, in run                                                          
		yielded = self.gen.send(value)     
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 386, in _wait_until_running                                                                  
		raise ValueError("Worker not started because status {} != 'starting'".format(self.status))                                                                
	ValueError: Worker not started because status stopped != 'starting'            

Note I printed out that `status` == stopped.

The problem is, this exception seems swallowed by `distributed` so that I can't react within my application, which hangs. Not sure where it's swallowed. Any way to pass it up?

This may be resolved in #1331 
Will check, thx

Just saw the problem recur, after #1331. As I'm starting via  `LocalCluster`, I don't think it uses the code path in `cli/dask_worker.py` that changed. My stacktrace looks like the following:

	tornado.application - ERROR - Exception in Future <tornado.concurrent.Future object at 0x7f16f962d5f8> after timeout                                          
	Traceback (most recent call last):     
	  File "lib/python3.5/site-packages/tornado/gen.py", line 910, in error_callback                                                
		future.result()                    
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "lib/python3.5/site-packages/distributed-1.18.0.6+11.ge94ba84c.dirty-py3.5.egg/distributed/nanny.py", line 300, in start 
		yield self._wait_until_running()   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1069, in run                                                          
		yielded = self.gen.send(value)     
	  File "lib/python3.5/site-packages/distributed-1.18.0.6+11.ge94ba84c.dirty-py3.5.egg/distributed/nanny.py", line 386, in _wait_until_running
		raise ValueError("Worker not started")                                     
	ValueError: Worker not started         


Anecdotally, this problem occurred while I also had a separate dask-scheduler process running on the same machine (controlling a separate cluster), so I wonder if the LocalCluster call couldn't start b/c the dask-scheduler process was holding some resource.
Running multiple schedulers on the same machine (or even same
process/thread) is pretty typical.  I don't expect this to be the problem.

On Tue, Aug 15, 2017 at 12:40 PM, Adam Klein <notifications@github.com>
wrote:

> Anecdotally, this problem occurred while I also had a separate
> dask-scheduler process running on the same machine (controlling a separate
> cluster), so I wonder if the LocalCluster call couldn't start b/c the
> dask-scheduler process was holding some resource.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1321#issuecomment-322520535>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszF4xHRT5lYUh5vGUSdZXXtJ2SK7eks5sYcoRgaJpZM4OxVIW>
> .
>

So... you can simulate the hanging behavior by adding a `import time; time.sleep(60)` here - https://github.com/dask/distributed/blob/master/distributed/worker.py#L265

This doesn't get to the root of why things are timing out, but it does show the hanging behavior that it cannot recover from this situation.
Unfortunately introducing that code doesn't cause things to fail for me. 
Interesting. All I have to do is:

     from distributed import LocalCluster
     LocalCluster() 
     # this is never reached, main process is hung

and I get the stack trace above (from all my workers)

Also, when I Ctrl-C to interrupt, things are here:

	/home/aklein/src/distributed/distributed/deploy/local.py in __init__(self, n_workers, threads_per_worker, processes, loop, start, ip, scheduler_port, silence_logs, diagnostics_port, services, worker_services, nanny, **worker_kwargs)
		114                                
		115         if start:              
	--> 116             sync(self.loop, self._start, ip)                          
		117                                
		118         clusters_to_close.add(self)                                   

	/home/aklein/src/distributed/distributed/utils.py in sync(loop, func, *args, **kwargs)                                                                       
		235     loop.add_callback(f)       
		236     while not e.is_set():      
	--> 237         e.wait(1000000)        
		238     if error[0]:               
		239         six.reraise(*error[0]) 

	lib/python3.5/threading.py in wait(self, timeout)                                                                              
		547             signaled = self._flag                                     
		548             if not signaled:   
	--> 549                 signaled = self._cond.wait(timeout)                   
		550             return signaled    
		551                                

	lib/python3.5/threading.py in wait(self, timeout)                                                                              
		295             else:              
		296                 if timeout > 0:
	--> 297                     gotit = waiter.acquire(True, timeout)             
		298                 else:          
		299                     gotit = waiter.acquire(False)                     

Reproduced.  It looks like death_timeout is set to 60.  You can increase this time in the following way:

    LocalCluster(death_timeout=120)

Resolving patch:
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -174,9 +174,13 @@ def _start_worker(self, port=0, processes=None, death_timeout=60, **kwargs):
 
         self.workers.append(w)
 
-        while w.worker_address not in self.scheduler.worker_info:
+        while w.status != 'closed' and w.worker_address not in self.scheduler.worker_info:
             yield gen.sleep(0.01)
 
+        if w.status == 'closed':
+            self.workers.remove(w)
+            raise gen.TimeoutError("Worker failed to start")
+
         raise gen.Return(w)
 
     def start_worker(self, ncores=0, **kwargs):
@@ -249,7 +253,10 @@ def close(self):
             try:
                 self._thread.join(timeout=1)
             finally:
-                self.loop.close()
+                try:
+                    self.loop.close()
+                except ValueError:
+                    pass
             del self._thread
 
     @gen.coroutine


Test patch:
diff --git a/distributed/deploy/tests/test_local.py b/distributed/deploy/tests/test_local.py
--- a/distributed/deploy/tests/test_local.py
+++ b/distributed/deploy/tests/test_local.py
@@ -311,3 +311,11 @@ def test_memory_nanny(loop):
             info = c.scheduler_info()
             assert (sum(w['memory_limit'] for w in info['workers'].values())
                     < TOTAL_MEMORY * 0.9)
+
+
+def test_death_timeout_raises(loop):
+    with pytest.raises(gen.TimeoutError):
+        with LocalCluster(scheduler_port=0, silence_logs=False,
+                          death_timeout=1e-10, diagnostics_port=None,
+                          loop=loop) as cluster:
+            pass