Repository: dask/distributed

Issue:
Failed to start workers on 2nd platform after upgrading to latest GIT status
After upgrading all our nodes running dask to the latest GIT status (needed for retry mechanism) we encountered the problem shown by the log below when starting up the workers on the 2nd host... (First host is fine!)

Unfortunately I cannot help a lot more, because I'm not the administrator of the cluster. After IT rolled back to 1.20.2 the workers on all nodes startup fine...

```
dask-worker tcp://<IP_ADDRESS>:8786 --memory-limit 6e9 --local-directory /u01/USERNAME/dask_worker_dir --nprocs 8 --nthreads 1

distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:24534'
distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:17988'
distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:24278'
distributed.diskutils - WARNING - Found stale lock file and directory u'/u01/USERNAME/dask_worker_dir/worker-HyyuR7', purging
distributed.diskutils - WARNING - Found stale lock file and directory u'/u01/USERNAME/dask_worker_dir/worker-0B5Jok', purging
distributed.diskutils - WARNING - Found stale lock file and directory u'/u01/USERNAME/dask_worker_dir/worker-NmuI4k', purging
distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:18293'
distributed.worker - INFO -       Start worker at:  tcp://<IP_ADDRESS>:37222
distributed.worker - INFO -          Listening to:  tcp://<IP_ADDRESS>:37222
distributed.worker - INFO -              nanny at:        <IP_ADDRESS>:24534
distributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                    6.00 GB
distributed.worker - INFO -       Local Directory: /u01/USERNAME/dask_worker_dir/worker-XH_Qpm
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:12528'
distributed.worker - INFO -       Start worker at:  tcp://<IP_ADDRESS>:16643
distributed.worker - INFO -          Listening to:  tcp://<IP_ADDRESS>:16643
distributed.worker - INFO -              nanny at:        <IP_ADDRESS>:17988
distributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                    6.00 GB
distributed.worker - INFO -       Local Directory: /u01/USERNAME/dask_worker_dir/worker-eUNonW
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:35961'
distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/distributed/nanny.py", line 522, in run
    yield worker._start(*worker_start_args)
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/distributed/worker.py", line 372, in _start
    yield self._register_with_scheduler()
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/distributed/worker.py", line 295, in _register_with_scheduler
    (response,))
ValueError: Unexpected response from register: {'status': 'error', 'message': u'name taken, -0', 'time': 1519054893.084648}
distributed.worker - INFO -       Start worker at:  tcp://<IP_ADDRESS>:36269
distributed.worker - INFO -          Listening to:  tcp://<IP_ADDRESS>:36269
distributed.worker - INFO -              nanny at:        <IP_ADDRESS>:24278
distributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                    6.00 GB
distributed.worker - INFO -       Local Directory: /u01/USERNAME/dask_worker_dir/worker-3hke9z
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - ERROR - Failed to start worker


```


Discussion:
> ValueError: Unexpected response from register: {'status': 'error', 'message': u'name taken, -0', 'time': 1519054893.084648}

Yeah, that's interesting and definitely a bug.  Short term I recommend not using the `--nprocs` parameter, and instead just starting many processes.

Thanks for the option.

I did try "--nprocs = 1" yesterday on the 2nd node. However I saw strange numbers in "worker count" and available GB coming from the client.restart(). I didn't trust the environment anymore so I had IT roll back the install. Drawback of working for large company...

I meant just don't use --nprocs

On Tue, Feb 20, 2018 at 4:27 AM, BnY <notifications@github.com> wrote:

> Thanks for the option.
>
> I did try "--nprocs = 1" yesterday on the 2nd node. However I saw strange
> numbers in "worker count" and available GB coming from the
> client.restart(). I didn't trust the environment anymore so I had IT roll
> back the install. Drawback of working for large company...
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1772#issuecomment-366918298>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszHj9QaiGvcsUJ4hWbIVHdIGFuyO5ks5tWpANgaJpZM4SK0_4>
> .
>


Resolving patch:
diff --git a/distributed/cli/dask_worker.py b/distributed/cli/dask_worker.py
--- a/distributed/cli/dask_worker.py
+++ b/distributed/cli/dask_worker.py
@@ -217,7 +217,7 @@ def del_pid_file():
                  local_dir=local_directory, death_timeout=death_timeout,
                  preload=preload, preload_argv=preload_argv,
                  security=sec, contact_address=contact_address,
-                 name=name if nprocs == 1 else name + '-' + str(i),
+                 name=name if nprocs == 1 or not name else name + '-' + str(i),
                  **kwargs)
                for i in range(nprocs)]
 


Test patch:
diff --git a/distributed/cli/tests/test_dask_worker.py b/distributed/cli/tests/test_dask_worker.py
--- a/distributed/cli/tests/test_dask_worker.py
+++ b/distributed/cli/tests/test_dask_worker.py
@@ -120,16 +120,18 @@ def test_nprocs_expands_name(loop):
     with popen(['dask-scheduler', '--no-bokeh']) as sched:
         with popen(['dask-worker', '127.0.0.1:8786', '--nprocs', '2',
                     '--name', 'foo']) as worker:
-            with Client('tcp://127.0.0.1:8786', loop=loop) as c:
-                start = time()
-                while len(c.scheduler_info()['workers']) < 2:
-                    sleep(0.2)
-                    assert time() < start + 10
+            with popen(['dask-worker', '127.0.0.1:8786', '--nprocs', '2']) as worker:
+                with Client('tcp://127.0.0.1:8786', loop=loop) as c:
+                    start = time()
+                    while len(c.scheduler_info()['workers']) < 4:
+                        sleep(0.2)
+                        assert time() < start + 10
 
-                info = c.scheduler_info()
-                names = [d['name'] for d in info['workers'].values()]
-                assert all(name.startswith('foo') for name in names)
-                assert len(set(names)) == 2
+                    info = c.scheduler_info()
+                    names = [d['name'] for d in info['workers'].values()]
+                    foos = [n for n in names if n.startswith('foo')]
+                    assert len(foos) == 2
+                    assert len(set(names)) == 4
 
 
 @pytest.mark.skipif(not sys.platform.startswith('linux'),