Repository: dask/distributed

Issue:
ValueError / hang on LocalCluster startup
I'm loading my conda environment off NAS storage, and when usage is heavy, there can be long delays in loading things, which I believe can exceed the default timeout (five seconds) here:

https://github.com/dask/distributed/blob/master/distributed/client.py#L428

As a result I see this on a call to `LocalCluster()`:

	Traceback (most recent call last):                                                                                                                            
	  File "lib/python3.5/site-packages/tornado/gen.py", line 910, in error_callback                                                
		future.result()                                                                                                                                           
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)                                                                                                                            
	  File "<string>", line 4, in raise_exc_info                                                                                                                  
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "lib/python3.5/site-packages/distributed/nanny.py", line 300, in start
		yield self._wait_until_running()                                           
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()                                                                                                                                   
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
		raise_exc_info(self._exc_info)                                             
	  File "<string>", line 4, in raise_exc_info                                                                                                                  
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1069, in run
		yielded = self.gen.send(value)                                                                                                                            
	  File "lib/python3.5/site-packages/distributed/nanny.py", line 386, in _wait_until_running
		raise ValueError("Worker not started")                                                                                                                                                                                                                                                                                   
	ValueError: Worker not started         

This `ValueError` does not seem to be passed up to my application; my application seems frozen waiting for the cluster start up (not 100% sure where) - ie, it never returns from the `LocalCluster()` call. Although the multiprocessing pool actually does start up, in this case too late to matter.

How can I catch / retry this situation?


Discussion:
Or, it could have nothing to do with a timeout. :) I see the guard has to do with the status, and there's an iffy looking `delay = 0.05` on line 383 of nanny.py ...

EDIT:  hmm, I'll try to see what the `self.status` is instead of `starting`, I guess that's the key here ...
Ok, I seem to have induced a ValueError with a indistriminately sent SIGINT (Ctrl-C) which shows up as the following:

	 File "lib/python3.5/multiprocessing/forkserver.py", line 164, in main        
		rfds = [key.fileobj for (key, events) in selector.select()]                                                                                               
	  File "lib/python3.5/selectors.py", line 445, in select
		fd_event_list = self._epoll.poll(timeout, max_ev)                                                                                                         
	KeyboardInterrupt                                                                             

	tornado.application - ERROR - Multiple exceptions in yield list                
	Traceback (most recent call last):     
	  File "lib/python3.5/site-packages/tornado/gen.py", line 828, in callback                                                      
		result_list.append(f.result())     
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/deploy/local.py", line 173, in _start_worker                                                                 
		yield w._start()                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 135, in _start                                                                               
		response = yield self.instantiate()                                        
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 189, in instantiate                                                                          
		self.process.start())              
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1063, in run                                                          
		yielded = self.gen.throw(*exc_info)                                        
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 300, in start 
		yield self._wait_until_running()   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1055, in run                                                          
		value = future.result()            
	  File "lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result                                                 
		raise_exc_info(self._exc_info)     
	  File "<string>", line 4, in raise_exc_info                                   
	  File "lib/python3.5/site-packages/tornado/gen.py", line 1069, in run                                                          
		yielded = self.gen.send(value)     
	  File "/home/aklein/src/distributed/distributed/nanny.py", line 386, in _wait_until_running                                                                  
		raise ValueError("Worker not started because status {} != 'starting'".format(self.status))                                                                
	ValueError: Worker not started because status stopped != 'starting'            

Note I printed out that `status` == stopped.

The problem is, this exception seems swallowed by `distributed` so that I can't react within my application, which hangs. Not sure where it's swallowed. Any way to pass it up?


Resolving patch:
diff --git a/distributed/cli/dask_worker.py b/distributed/cli/dask_worker.py
--- a/distributed/cli/dask_worker.py
+++ b/distributed/cli/dask_worker.py
@@ -211,11 +211,9 @@ def handle_signal(signum, frame):
     signal.signal(signal.SIGINT, handle_signal)
     signal.signal(signal.SIGTERM, handle_signal)
 
-    for n in nannies:
-        n.start(addr)
-
     @gen.coroutine
     def run():
+        yield [n.start(addr) for n in nannies]
         while all(n.status != 'closed' for n in nannies):
             yield gen.sleep(0.2)
 


Test patch: