Repository: dask/distributed

Issue:
The worker_kwargs argument doesn't work for LocalCluster
Running this:

```
cluster = dask.distributed.LocalCluster(
    scheduler_port=0,
    dashboard_address=":0",
    n_workers=1,
    worker_kwargs={}
)
```

Results in this complaint:

```
Process Dask Worker process (from Nanny):
Traceback (most recent call last):
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/nix/store/8zmsa97jhnns0p96ajqq20lan2hxbh7r-python3.7-distributed-2.3.0/lib/python3.7/site-packages/distributed/process.py", line 179, in _run
    target(*args, **kwargs)
  File "/nix/store/8zmsa97jhnns0p96ajqq20lan2hxbh7r-python3.7-distributed-2.3.0/lib/python3.7/site-packages/distributed/nanny.py", line 636, in _run
    worker = Worker(**worker_kwargs)
  File "/nix/store/8zmsa97jhnns0p96ajqq20lan2hxbh7r-python3.7-distributed-2.3.0/lib/python3.7/site-packages/distributed/worker.py", line 626, in __init__
    **kwargs
TypeError: __init__() got an unexpected keyword argument 'worker_kwargs'
```

The `worker_kwargs` is documented in the API and is meant to be passed to the Worker class.


Discussion:
I'm looking to use this to assign some artificial resources to the local cluster.
Thanks for reporting @CMCDragonkai. `worker_kwargs` is not meant to be a dictionary, but rather that all remaining keyword arguments will be passed to the `Worker` class. So, for example, if you wanted to pass `resources={"GPU": 2}` to the cluster workers you would do:

```python
cluster = dask.distributed.LocalCluster(
    scheduler_port=0,
    dashboard_address=":0",
    n_workers=1,
    resources={"GPU": 2}
)
```
I actually tried that as well, but it also didn't work.

On 4 February 2020 02:46:34 GMT+11:00, James Bourbeau <notifications@github.com> wrote:
>Thanks for reporting @CMCDragonkai. `worker_kwargs` is not meant to be
>a dictionary, but rather that all remaining keyword arguments will be
>passed to the `Worker` class. So, for example, if you wanted to pass
>`resources={"GPU": 2}` to the cluster workers you would do:
>
>```python
>cluster = dask.distributed.LocalCluster(
>    scheduler_port=0,
>    dashboard_address=":0",
>    n_workers=1,
>    resources={"GPU": 2}
>)
>```
>
>-- 
>You are receiving this because you were mentioned.
>Reply to this email directly or view it on GitHub:
>https://github.com/dask/distributed/issues/3435#issuecomment-581477544

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Do you get a similar output?

```python
In [1]: import dask.distributed

In [2]: cluster = dask.distributed.LocalCluster(
   ...:     scheduler_port=0,
   ...:     dashboard_address=":0",
   ...:     n_workers=1,
   ...:     resources={"GPU": 2}
   ...: )

In [3]: cluster.workers
Out[3]: {0: <Nanny: tcp://127.0.0.1:53111, threads: 4>}

In [4]: cluster.workers[0].resources
Out[4]: {'GPU': 2}
```
Ok it works. Nice the docs should be updated.

On a side-note, I actually wanted to use 2 local clusters. one local cluster having a certain set of resources, the other local cluster having a different set of resources, but it seems that the local cluster both will have their own schedulers. Wondering if there's a way to share a scheduler between 2 local clusters?

Resolving patch:
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -49,8 +49,6 @@ class LocalCluster(SpecCluster):
     asynchronous: bool (False by default)
         Set to True if using this cluster within async/await functions or within
         Tornado gen.coroutines.  This should remain False for normal use.
-    worker_kwargs: dict
-        Extra worker arguments, will be passed to the Worker constructor.
     blocked_handlers: List[str]
         A list of strings specifying a blacklist of handlers to disallow on the Scheduler,
         like ``['feed', 'run_function']``
@@ -68,6 +66,9 @@ class LocalCluster(SpecCluster):
         Network interface to use.  Defaults to lo/localhost
     worker_class: Worker
         Worker class used to instantiate workers from.
+    **worker_kwargs:
+        Extra worker arguments. Any additional keyword arguments will be passed
+        to the ``Worker`` class constructor.
 
     Examples
     --------


Test patch: