internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/distributed

Issue:
Small logging error
looks like you don't want parens around the args. (distributed version 1.21.8)

--- Logging error ---
Traceback (most recent call last):
  File " envs/a2/lib/python3.5/logging/__init__.py", line 986, in emit
    msg = self.format(record)
  File " envs/a2/lib/python3.5/logging/__init__.py", line 836, in format
    return fmt.format(record)
  File " envs/a2/lib/python3.5/logging/__init__.py", line 573, in format
    record.message = record.getMessage()
  File " envs/a2/lib/python3.5/logging/__init__.py", line 336, in getMessage
    msg = msg % self.args
TypeError: not enough arguments for format string
Call stack:
  File "<string>", line 1, in <module>
  File " envs/a2/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File " envs/a2/lib/python3.5/multiprocessing/spawn.py", line 119, in _main
    return self._bootstrap()
  File " envs/a2/lib/python3.5/multiprocessing/process.py", line 252, in _bootstrap
    self.run()
  File " envs/a2/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File " envs/a2/lib/python3.5/site-packages/distributed/process.py", line 174, in _run
    target(*args, **kwargs)
  File " envs/a2/lib/python3.5/site-packages/distributed/nanny.py", line 556, in _run
    loop.run_sync(run)
  File " envs/a2/lib/python3.5/site-packages/tornado/ioloop.py", line 453, in run_sync
    self.start()
  File " envs/a2/lib/python3.5/site-packages/tornado/ioloop.py", line 832, in start
    self._run_callback(self._callbacks.popleft())
  File " envs/a2/lib/python3.5/site-packages/tornado/ioloop.py", line 605, in _run_callback
    ret = callback()
  File " envs/a2/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File " envs/a2/lib/python3.5/site-packages/tornado/gen.py", line 1152, in inner
    self.run()
  File " envs/a2/lib/python3.5/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File " envs/a2/lib/python3.5/site-packages/distributed/client.py", line 1418, in _gather
    log("Couldn't gather %s keys, rescheduling %s", (len(response['keys']), response['keys']))
Message: "Couldn't gather %s keys, rescheduling %s"



Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1449,7 +1449,7 @@ def wait(k):
 
             if response['status'] == 'error':
                 log = logger.warning if errors == 'raise' else logger.debug
-                log("Couldn't gather %s keys, rescheduling %s", (len(response['keys']), response['keys']))
+                log("Couldn't gather %s keys, rescheduling %s", len(response['keys']), response['keys'])
                 for key in response['keys']:
                     self._send_to_scheduler({'op': 'report-key',
                                              'key': key})


Test patch:
Repository: dask/distributed

Issue:
Journey of a Task
This is a question on the current behavior. 
For distributed DASK setup with dask scheduler and dask workers running on different hosts and a client connected from a different host, does the the client receive the task result (stored in a dask worker) via dask scheduler ? 
Or the scheduler asks the client to get the task result directly from worker ? 



Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -489,8 +489,8 @@ class resembles executors in ``concurrent.futures`` but also allows
         Gives the client a name that will be included in logs generated on
         the scheduler for matters relating to this client
     direct_to_workers: bool (optional)
-        Can this client connect directly to workers or should it proxy through
-        the scheduler?
+        Whether or not to connect directly to the workers, or to ask
+        the scheduler to serve as intermediary.
     heartbeat_interval: int
         Time in milliseconds between heartbeats to scheduler
 
@@ -1609,6 +1609,10 @@ def gather(self, futures, errors='raise', maxsize=0, direct=None,
         errors: string
             Either 'raise' or 'skip' if we should raise if a future has erred
             or skip its inclusion in the output collection
+        direct: boolean
+            Whether or not to connect directly to the workers, or to ask
+            the scheduler to serve as intermediary.  This can also be set when
+            creating the Client.
         maxsize: int
             If the input is a queue then this produces an output queue with a
             maximum size.
@@ -1795,9 +1799,9 @@ def scatter(self, data, workers=None, broadcast=False, direct=None,
             Whether to send each data element to all workers.
             By default we round-robin based on number of cores.
         direct: bool (defaults to automatically check)
-            Send data directly to workers, bypassing the central scheduler
-            This avoids burdening the scheduler but assumes that the client is
-            able to talk directly with the workers.
+            Whether or not to connect directly to the workers, or to ask
+            the scheduler to serve as intermediary.  This can also be set when
+            creating the Client.
         maxsize: int (optional)
             Maximum size of queue if using queues, 0 implies infinite
         hash: bool (optional)
@@ -2295,7 +2299,9 @@ def get(self, dsk, keys, restrictions=None, loose_restrictions=None,
         sync: bool (optional)
             Returns Futures if False or concrete values if True (default).
         direct: bool
-            Gather results directly from workers
+            Whether or not to connect directly to the workers, or to ask
+            the scheduler to serve as intermediary.  This can also be set when
+            creating the Client.
 
         Examples
         --------


Test patch:
Repository: dask/distributed

Issue:
deploying with dask-ssh fails on distributed 1.27.1
After updating to distributed 1.27.1 I cannot deploy using dask-ssh. 

Here I report part of the output of running "dask-ssh localhost" on the terminal. I believe one fo the scripts (maybe deploy/ssh.py) is somehow being passed None in place of "distributed.cli.dask_worker". 

Deploying the scheduler and workers individually with dask-scheduler and dask-worker still works for me. 

> --------------------------------------------------------------- 
> 
>                  Dask.distributed v1.27.1
> 
> Worker nodes:
>   0: localhost
> 
> scheduler node: localhost:8786
> --------------------------------------------------------------- 
> 
> 
> [ worker localhost ] : /home/xyz/anaconda2/bin/python -m None localhost:8786 --nthreads 0 --host localhost --memory-limit auto
> [ scheduler localhost:8786 ] : /home/xyz/anaconda2/bin/python -m distributed.cli.dask_scheduler --port 8786
> Exception in thread Thread-2:
> Traceback (most recent call last):
> ...
> [ scheduler localhost:8786 ] : distributed.scheduler - INFO - -----------------------------------------------
> [ worker localhost ] : /home/xyz/anaconda2/bin/python: No module named None
> [ worker localhost ] : remote process exited with exit status 1




Discussion:
I don't yet know why the default changed, but the following does work
```
dask-ssh localhost --remote-dask-worker distributed.cli.dask_worker
```
in c94cde1f7f5bc06ae57a303f7345635e323dad21 the default was set to `None`, where previously this could not be changed at all.

-EDIT- sorry, that was the previous good commit. change was in d6ba683a55816ece13f66a68f7e2ef7668775800

Resolving patch:
diff --git a/distributed/cli/dask_ssh.py b/distributed/cli/dask_ssh.py
--- a/distributed/cli/dask_ssh.py
+++ b/distributed/cli/dask_ssh.py
@@ -96,7 +96,7 @@
 )
 @click.option(
     "--remote-dask-worker",
-    default=None,
+    default="distributed.cli.dask_worker",
     type=str,
     help="Worker to run. Defaults to distributed.cli.dask_worker",
 )


Test patch:
Repository: dask/distributed

Issue:
BokehRuntimeWarnings after #2747
After https://github.com/dask/distributed/pull/2747, I see warnings like

```
BokehUserWarning: ColumnDataSource's columns must be of the same length. Current lengths: ('address', 4), ('dashboard_host', 4), ('dashboard_port', 4), ('nbytes', 4), ('nbytes-color', 4), ('nbytes-half', 4), ('nbytes_text', 4), ('nprocessing', 4), ('nprocessing-color', 4), ('nprocessing-half', 4), ('worker', 2), ('y', 4)
BokehUserWarning: ColumnDataSource's columns must be of the same length. Current lengths: ('address', 4), ('dashboard_host', 4), ('dashboard_port', 4), ('nbytes', 4), ('nbytes-color', 4), ('nbytes-half', 4), ('nbytes_text', 4), ('nprocessing', 4), ('nprocessing-color', 4), ('nprocessing-half', 4), ('worker', 2), ('y', 4)
```

probably something with https://github.com/dask/distributed/pull/2747/files#diff-bb0ae02b02c7f964b31a044e6dc83b19L1336


Discussion:
I'll take a look.  Thanks for raising Tom.

On Thu, Jun 6, 2019 at 10:01 AM Tom Augspurger <notifications@github.com>
wrote:

> After #2747 <https://github.com/dask/distributed/pull/2747>, I see
> warnings like
>
> BokehUserWarning: ColumnDataSource's columns must be of the same length. Current lengths: ('address', 4), ('dashboard_host', 4), ('dashboard_port', 4), ('nbytes', 4), ('nbytes-color', 4), ('nbytes-half', 4), ('nbytes_text', 4), ('nprocessing', 4), ('nprocessing-color', 4), ('nprocessing-half', 4), ('worker', 2), ('y', 4)
> BokehUserWarning: ColumnDataSource's columns must be of the same length. Current lengths: ('address', 4), ('dashboard_host', 4), ('dashboard_port', 4), ('nbytes', 4), ('nbytes-color', 4), ('nbytes-half', 4), ('nbytes_text', 4), ('nprocessing', 4), ('nprocessing-color', 4), ('nprocessing-half', 4), ('worker', 2), ('y', 4)
>
> probably something with
> https://github.com/dask/distributed/pull/2747/files#diff-bb0ae02b02c7f964b31a044e6dc83b19L1336
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2754?email_source=notifications&email_token=AACKZTHGHVLHUBJZEOFB27LPZE7GPA5CNFSM4HVFHTDKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GYCML3A>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACKZTG36WLAU4NTSSVG7D3PZE7GPANCNFSM4HVFHTDA>
> .
>


Resolving patch:
diff --git a/distributed/dashboard/scheduler.py b/distributed/dashboard/scheduler.py
--- a/distributed/dashboard/scheduler.py
+++ b/distributed/dashboard/scheduler.py
@@ -449,7 +449,7 @@ def update(self):
                     "nbytes_text": nbytes_text,
                     "dashboard_host": dashboard_host,
                     "dashboard_port": dashboard_port,
-                    "address": [ws.address for ws in workers],
+                    "worker": [ws.address for ws in workers],
                     "y": y,
                 }
 


Test patch:
Repository: dask/distributed

Issue:
SpecCluster error when removing workers
When attempting to scale down a `SpecCluster` I'm getting the following error.

```python-traceback
distributed.utils - ERROR - 'Worker' object has no attribute 'worker_address'
Traceback (most recent call last):
  File "/home/nfs/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 714, in log_errors
    yield
  File "/home/nfs/jtomlinson/Projects/dask/distributed/distributed/deploy/adaptive.py", line 176, in _retire_workers
    await f
  File "/home/nfs/jtomlinson/Projects/dask/distributed/distributed/deploy/spec.py", line 324, in scale_down
    if v.worker_address in workers:
AttributeError: 'Worker' object has no attribute 'worker_address'
```


Discussion:

Resolving patch:
diff --git a/distributed/deploy/spec.py b/distributed/deploy/spec.py
--- a/distributed/deploy/spec.py
+++ b/distributed/deploy/spec.py
@@ -318,8 +318,6 @@ def new_worker_spec(self):
     async def scale_down(self, workers):
         workers = set(workers)
 
-        # TODO: this is linear cost.  We should be indexing by name or something
-        to_close = [w for w in self.workers.values() if w.address in workers]
         for k, v in self.workers.items():
             if v.worker_address in workers:
                 del self.worker_spec[k]


Test patch:
Repository: dask/distributed

Issue:
[DOC] LocalCluster docs still refer to bokeh, not dashboard
https://distributed.dask.org/en/latest/local-cluster.html still has this example at the bottom:
```python
>>> LocalCluster(service_kwargs={'bokeh': {'prefix': '/foo'}})  # doctest: +SKIP
```
which doesn't work, but changing 'bokeh' to 'dashboard' fixes it.

Apologies for not supplying a PR at this point - I'm about to go on holiday and wanted to file this before I forgot about it.


Discussion:

Resolving patch:
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -80,7 +80,7 @@ class LocalCluster(SpecCluster):
 
     Pass extra keyword arguments to Bokeh
 
-    >>> LocalCluster(service_kwargs={'bokeh': {'prefix': '/foo'}})  # doctest: +SKIP
+    >>> LocalCluster(service_kwargs={'dashboard': {'prefix': '/foo'}})  # doctest: +SKIP
     """
 
     def __init__(


Test patch:
Repository: dask/distributed

Issue:
SpecCluster Multiple IP addresses
There are situations when creating a distributed cluster where the workers will need to connect to the scheduler on an internal private IP address but the client should connect on an external public IP.

One example of this would be out-of-cluster kubernetes where the workers need to connect via the internal container networking IP but the client should connect through an external load balancer or node port. Another would be on AWS ECS where the scheduler is assigned both an internal and external address, sadly workers are not able to route to the external address and clients are not able to route to the internal address.

When creating the scheduler via the various APIs both addresses will be known, but currently the `SpecCluster` only takes a single address which is passed on to the worker tasks as well as used to connect the RPC.

I propose that we make it possible for an external address to optionally also be set, which will be used for the RPC and daskboard urls. The `scheduler_comm` is created [here](https://github.com/dask/distributed/blob/master/distributed/deploy/spec.py#L228) and we could change it to `self.scheduler.external_address or self.scheduler.address` with `self.scheduler.external_address` defaulting to `None`.


Discussion:

Resolving patch:
diff --git a/distributed/deploy/spec.py b/distributed/deploy/spec.py
--- a/distributed/deploy/spec.py
+++ b/distributed/deploy/spec.py
@@ -23,6 +23,7 @@ class ProcessInterface:
 
     def __init__(self):
         self.address = None
+        self.external_address = None
         self.lock = asyncio.Lock()
         self.status = "created"
 
@@ -225,7 +226,7 @@ async def _start(self):
         self.status = "starting"
         self.scheduler = await self.scheduler
         self.scheduler_comm = rpc(
-            self.scheduler.address,
+            getattr(self.scheduler, "external_address", None) or self.scheduler.address,
             connection_args=self.security.get_connection_args("client"),
         )
         await super()._start()


Test patch:
Repository: dask/distributed

Issue:
dask.distributed.progress no longer callable in 2.3.0?
We've used the progress() function from dask.distributed a bunch in the past to display a progress bar in JupyterLab, but it seems to have stopped working after upgrading to Dask 2.3.0:

```
from dask.distributed import Client, progress
import dask.dataframe as dd

df = dd.demo.make_timeseries('2010', '2016',
                             {'value': float, 'name': str, 'id': int},
                             freq='10s', partition_freq='7d', seed=1)

df = df.persist()
progress(df)
```

Executing this in a single cell in JupyterLab (with an existing Dask cluster already running) results in:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-16af814d7204> in <module>
      7 
      8 df = df.persist()
----> 9 progress(df)

TypeError: 'module' object is not callable
```

Let me know if I can provide any more info. Thanks!


Discussion:

Resolving patch:
diff --git a/distributed/__init__.py b/distributed/__init__.py
--- a/distributed/__init__.py
+++ b/distributed/__init__.py
@@ -3,7 +3,7 @@
 from .actor import Actor, ActorFuture
 from .core import connect, rpc
 from .deploy import LocalCluster, Adaptive, SpecCluster
-from .diagnostics import progress
+from .diagnostics.progressbar import progress
 from .client import (
     Client,
     Executor,


Test patch:
Repository: dask/distributed

Issue:
preload config option not working with nanny process
If starting worker processes via the Nanny, the config file option `distributed.worker.preload` is not honored by the worker processes.


Discussion:

Resolving patch:
diff --git a/distributed/nanny.py b/distributed/nanny.py
--- a/distributed/nanny.py
+++ b/distributed/nanny.py
@@ -75,8 +75,8 @@ def __init__(
         resources=None,
         silence_logs=None,
         death_timeout=None,
-        preload=(),
-        preload_argv=[],
+        preload=None,
+        preload_argv=None,
         security=None,
         contact_address=None,
         listen_address=None,
@@ -116,7 +116,11 @@ def __init__(
         self.resources = resources
         self.death_timeout = parse_timedelta(death_timeout)
         self.preload = preload
+        if self.preload is None:
+            self.preload = dask.config.get("distributed.worker.preload")
         self.preload_argv = preload_argv
+        if self.preload_argv is None:
+            self.preload_argv = dask.config.get("distributed.worker.preload-argv")
         self.Worker = Worker if worker_class is None else worker_class
         self.env = env or {}
         worker_kwargs.update(


Test patch:
diff --git a/distributed/tests/test_preload.py b/distributed/tests/test_preload.py
--- a/distributed/tests/test_preload.py
+++ b/distributed/tests/test_preload.py
@@ -1,12 +1,14 @@
 import os
+import pytest
 import shutil
 import sys
 import tempfile
 import pytest
 
-from distributed import Client, Scheduler, Worker
+import dask
+from distributed import Client, Scheduler, Worker, Nanny
 from distributed.utils_test import cluster
-from distributed.utils_test import cleanup, loop  # noqa F401
+from distributed.utils_test import loop, cleanup  # noqa F401
 
 
 PRELOAD_TEXT = """
@@ -55,6 +57,20 @@ def dask_setup(worker):
             assert w.foo == "setup"
 
 
+@pytest.mark.asyncio
+async def test_worker_preload_config(cleanup):
+    text = """
+def dask_setup(worker):
+    worker.foo = 'setup'
+"""
+    with dask.config.set({"distributed.worker.preload": text}):
+        async with Scheduler(port=0) as s:
+            async with Nanny(s.address) as w:
+                async with Client(s.address, asynchronous=True) as c:
+                    d = await c.run(lambda dask_worker: dask_worker.foo)
+                    assert d == {w.worker_address: "setup"}
+
+
 def test_worker_preload_module(loop):
     def check_worker():
         import worker_info
Repository: dask/distributed

Issue:
Stop Queue from printing it's name on __init__
There's a print statement here:

https://github.com/dask/distributed/blob/8d07bf162b1f6434fc982f36360c5cedbac369a6/distributed/queues.py#L50

Would be nice to replace it with logging.debug()


Discussion:
Whoops, thanks. Interested in submitting a PR for that? Otherwise, I'll submit one later today.

(note: it should probably be `logger.debug`, rather than `logging.debug`)
Sure, I will
Great, thanks @roveo! 

Resolving patch:
diff --git a/distributed/queues.py b/distributed/queues.py
--- a/distributed/queues.py
+++ b/distributed/queues.py
@@ -47,7 +47,7 @@ def __init__(self, scheduler):
         self.scheduler.extensions["queues"] = self
 
     def create(self, stream=None, name=None, client=None, maxsize=0):
-        print("name", name)
+        logger.debug("Queue name: {}".format(name))
         if name not in self.queues:
             self.queues[name] = tornado.queues.Queue(maxsize=maxsize)
             self.client_refcount[name] = 1


Test patch:
Repository: dask/distributed

Issue:
TaskPrefix is not JSON serializable on Graph page
With the Graph layout page open, this logs errors:

```python
import dask.array as da
from dask.distributed import Client

client = Client()
f = client.submit(lambda x: x + 1, 0)
```

Like

```
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type TaskPrefix is not JSON serializable
```

The relevant bits are probably

```
  File "/Users/taugspurger/sandbox/distributed/distributed/dashboard/components/scheduler.py", line 1192, in update
    self.add_new_nodes_edges(new, new_edges, update=update)
...
  File "/Users/taugspurger/sandbox/distributed/distributed/dashboard/components/scheduler.py", line 1243, in add_new_nodes_edges
    self.node_source.data.update(node)
...
```

cc @mrocklin 


Discussion:

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -1219,7 +1219,7 @@ def add_new_nodes_edges(self, new, new_edges, update=False):
                 node_x.append(xx)
                 node_y.append(yy)
                 node_state.append(task.state)
-                node_name.append(task.prefix)
+                node_name.append(task.prefix.name)
 
             for a, b in new_edges:
                 try:


Test patch:
diff --git a/distributed/dashboard/tests/test_scheduler_bokeh.py b/distributed/dashboard/tests/test_scheduler_bokeh.py
--- a/distributed/dashboard/tests/test_scheduler_bokeh.py
+++ b/distributed/dashboard/tests/test_scheduler_bokeh.py
@@ -442,6 +442,8 @@ def test_TaskGraph(c, s, a, b):
     gp.update()
     assert set(map(len, gp.node_source.data.values())) == {6}
     assert set(map(len, gp.edge_source.data.values())) == {5}
+    json.dumps(gp.edge_source.data)
+    json.dumps(gp.node_source.data)
 
     da = pytest.importorskip("dask.array")
     x = da.random.random((20, 20), chunks=(10, 10)).persist()
Repository: dask/distributed

Issue:
When --dashboard-prefix is used root location is always redirected to /status without adding the prefix
The problem arises when using --dashboard-prefix, using http://DASHBOARD[:PORT]/PREFIX/ redirects to http://DASHBOARD[:PORT]/status instead of http://DASHBOARD[:PORT]/PREFIX/status.


Discussion:

Resolving patch:
diff --git a/distributed/dashboard/core.py b/distributed/dashboard/core.py
--- a/distributed/dashboard/core.py
+++ b/distributed/dashboard/core.py
@@ -5,6 +5,7 @@
 import bokeh
 from bokeh.server.server import Server
 from tornado import web
+from urllib.parse import urljoin
 
 
 if LooseVersion(bokeh.__version__) < LooseVersion("0.13.0"):
@@ -34,7 +35,13 @@ def listen(self, addr):
                     check_unused_sessions_milliseconds=500,
                     allow_websocket_origin=["*"],
                     use_index=False,
-                    extra_patterns=[(r"/", web.RedirectHandler, {"url": "/status"})],
+                    extra_patterns=[
+                        (
+                            r"/",
+                            web.RedirectHandler,
+                            {"url": urljoin(self.prefix.rstrip("/") + "/", r"status")},
+                        )
+                    ],
                 )
                 server_kwargs.update(self.server_kwargs)
                 self.server = Server(self.apps, **server_kwargs)


Test patch: