internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/distributed

Issue:
Avoid is_coroutine_function calls in Server
We currently support handlers that are either synchronous or asynchronous 

https://github.com/dask/distributed/blob/1297b18ff09276f6ad1553d40ab3ce77acf0fc0e/distributed/core.py#L567-L572

In #4443 we learned that these checks take up some time.  In a recent run it takes 100ms out of about a 2.5s run, so around 4%

There are a couple of ways to address this:

1. We can try to find some other cheaper check
2. We can make all of our handlers `async def` functions and asyncify the entire codebase (we should also check that this doesn't incur other costs if so)


cc @dask/scheduler-performance 
Use iscoroutinefunction utility function in actor
Appears `actor` grabs `iscoroutinefunction` from `inspect` even though we have the utility function defined in `distributed.utils`. Maybe we should just use our utility function here as well

https://github.com/dask/distributed/blob/3407aa31be08ce100bd96e42acb99e1d7daf996e/distributed/actor.py#L3


Discussion:

Resolving patch:
diff --git a/distributed/actor.py b/distributed/actor.py
--- a/distributed/actor.py
+++ b/distributed/actor.py
@@ -1,12 +1,11 @@
 import asyncio
 import functools
-from inspect import iscoroutinefunction
 import threading
 from queue import Queue
 
 from .client import Future, default_client
 from .protocol import to_serialize
-from .utils import thread_state, sync
+from .utils import iscoroutinefunction, thread_state, sync
 from .utils_comm import WrappedKey
 from .worker import get_worker
 
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -1246,6 +1246,7 @@ def color_of(x, palette=palette):
     return palette[n % len(palette)]
 
 
+@functools.lru_cache(None)
 def iscoroutinefunction(f):
     return inspect.iscoroutinefunction(f) or gen.is_coroutine_function(f)
 
@@ -1343,8 +1344,7 @@ def parse_ports(port):
     return ports
 
 
-def is_coroutine_function(f):
-    return asyncio.iscoroutinefunction(f) or gen.is_coroutine_function(f)
+is_coroutine_function = iscoroutinefunction
 
 
 class Log(str):


Test patch:
Repository: dask/distributed

Issue:
System tab (scheduler resources) widget
Love this project! It's so much fun to watch all the widgets moving along while doing Dask work :) 

It would be helpful to have a widget for the /system page from the standalone dashboard

![image](https://user-images.githubusercontent.com/8964039/107394567-06813080-6aca-11eb-80d5-153e0e1c41ed.png)



Discussion:
Thanks for raising this @rikturr.

This change would need to be made in dask/distributed as the Jupyter Lab extension exposes plots from the distributed dashboard. I'll move this issue over there.

I agree it would probably be useful to have the scheduler system monitor as an option.

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -2184,6 +2184,16 @@ def individual_graph_doc(scheduler, extra, doc):
         doc.theme = BOKEH_THEME
 
 
+def individual_systemmonitor_doc(scheduler, extra, doc):
+    with log_errors():
+        sysmon = SystemMonitor(scheduler, sizing_mode="stretch_both")
+        doc.title = "Dask: Scheduler System Monitor"
+        add_periodic_callback(doc, sysmon, 500)
+
+        doc.add_root(sysmon.root)
+        doc.theme = BOKEH_THEME
+
+
 def individual_profile_doc(scheduler, extra, doc):
     with log_errors():
         prof = ProfileTimePlot(scheduler, sizing_mode="scale_width", doc=doc)
diff --git a/distributed/dashboard/scheduler.py b/distributed/dashboard/scheduler.py
--- a/distributed/dashboard/scheduler.py
+++ b/distributed/dashboard/scheduler.py
@@ -34,6 +34,7 @@
     individual_memory_by_key_doc,
     individual_compute_time_per_key_doc,
     individual_aggregate_time_per_action_doc,
+    individual_systemmonitor_doc,
 )
 from .worker import counters_doc
 from .components.nvml import gpu_memory_doc, gpu_utilization_doc  # noqa: 1708
@@ -90,4 +91,5 @@ def connect(application, http_server, scheduler, prefix=""):
     "/individual-aggregate-time-per-action": individual_aggregate_time_per_action_doc,
     "/individual-gpu-memory": gpu_memory_doc,
     "/individual-gpu-utilization": gpu_utilization_doc,
+    "/individual-scheduler-system": individual_systemmonitor_doc,
 }


Test patch:
Repository: dask/distributed

Issue:
Incorrect number of futures with client.map if both keys and batch_size are specified
If both `keys` and `batch_size` are specified, then the number of `futures` created with `client.map` is incorrect. Here's a reproducible example:

```python
from dask.distributed import Client
client = Client()
futs = client.map(lambda x: x**2, list(range(20)), batch_size=5, key=[f"{x}" for x in list(range(20))])
print(len(futs)) # prints 80
```

If you try to collect these futures with `client.gather` then client will hang indefinitely.

This has to do with simultaneously specifying `batch_size` and custom `key` list, removing either of them will generate the correct number of futures.

I suspect the problem is in line 1701 here: https://github.com/dask/distributed/blob/4ce8c65039e9b531878f9d77cf884920690a3745/distributed/client.py#L1692-L1716

Specifically, it seems that the same keys are passed to every batch, this is problematic if key is a list. My suggested fix for this is to modify it as follows (code removed, it was faulty, see PR for working code).

Would be happy to submit PR if this code makes sense.

**Environment**:

- Dask version: 2021.02.0
- Python version: 3.6
- Operating System: MacOS
- Install method (conda, pip, source): conda



Discussion:
Thanks for reporting @SultanOrazbayev. Batching the user-specified keys in this case seems like the right thing to do. A PR would be welcome 
Great, I'll work on it now.

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1693,6 +1693,10 @@ def map(
             batches = list(
                 zip(*[partition_all(batch_size, iterable) for iterable in iterables])
             )
+            if isinstance(key, list):
+                keys = [list(element) for element in partition_all(batch_size, key)]
+            else:
+                keys = [key for _ in range(len(batches))]
             return sum(
                 [
                     self.map(
@@ -1710,7 +1714,7 @@ def map(
                         pure=pure,
                         **kwargs,
                     )
-                    for batch in batches
+                    for key, batch in zip(keys, batches)
                 ],
                 [],
             )


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -229,6 +229,20 @@ async def test_map_batch_size(c, s, a, b):
     assert result == list(range(100, 120, 2))
 
 
+@gen_cluster(client=True)
+async def test_custom_key_with_batches(c, s, a, b):
+    """ Test of <https://github.com/dask/distributed/issues/4588>"""
+
+    futs = c.map(
+        lambda x: x ** 2,
+        range(10),
+        batch_size=5,
+        key=[str(x) for x in range(10)],
+    )
+    assert len(futs) == 10
+    await wait(futs)
+
+
 @gen_cluster(client=True)
 async def test_compute_retries(c, s, a, b):
     args = [ZeroDivisionError("one"), ZeroDivisionError("two"), 3]
Repository: dask/distributed

Issue:
Locks and LocalCluster
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
Using Locks with a LocalCluster generates RuntimeWarnings (does not seem to generate the same warnings with other kinds of cluster).

```
.../lib/python3.8/site-packages/distributed/client.py:4717: RuntimeWarning: coroutine 'wait_for' was never awaited
  c.close(timeout=3)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
.../lib/python3.8/site-packages/distributed/client.py:4717: RuntimeWarning: coroutine 'Client._close' was never awaited
  c.close(timeout=3)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
.../lib/python3.8/site-packages/distributed/client.py:4717: RuntimeWarning: coroutine 'wait_for' was never awaited
  c.close(timeout=3)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
```

**What you expected to happen**:
No warnings !

**Minimal Complete Verifiable Example**:

```python
import time

from distributed import Client, LocalCluster, Lock


def task(lock: Lock):
    with lock:
        time.sleep(1)


if __name__ == "__main__":
    cluster = LocalCluster()
    client = Client(cluster)

    lock_a = Lock(name="a", client=client)

    futures = []
    for i in range(2):
        f = client.submit(func=task, lock=lock_a)
        futures.append(f)

    client.gather(futures=futures)

    cluster.close()

```


**Environment**:

- Dask version: 2021.3.0
- Python version: 3.8.5
- Operating System: Ubuntu 16.04.7
- Install method (conda, pip, source): conda



Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -4726,7 +4726,10 @@ def _close_global_client():
     if c is not None:
         c._should_close_loop = False
         with suppress(TimeoutError, RuntimeError):
-            c.close(timeout=3)
+            if c.asynchronous:
+                c.loop.add_callback(c.close, timeout=3)
+            else:
+                c.close(timeout=3)
 
 
 atexit.register(_close_global_client)


Test patch:
Repository: dask/distributed

Issue:
DEBUG log messages imply UCX config options are invalid
When debugging is enabled, distributed outputs the following messages:
```
distributed.comm.ucx - DEBUG - Key: cuda_copy with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: tcp with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: nvlink with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: infiniband with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: rdmacm with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: net-devices with value: None not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: reuse-endpoints with value: None not a valid UCX configuration option.
```
When the corresponding env vars as set, such as `export DASK_UCX__CUDA_COPY=True`, the code in `ucx.py` [here](https://github.com/dask/distributed/blob/d41169bf231949f7dfb69e052a9690f7381e62f7/distributed/comm/ucx.py#L500) seems to behave correctly, but the check [here](https://github.com/dask/distributed/blob/d41169bf231949f7dfb69e052a9690f7381e62f7/distributed/comm/ucx.py#L517) still fails and prints the debug messages.

This can be seen simply by setting:
```
export DASK_LOGGING__DISTRIBUTED="DEBUG"
```
and running the scheduler like so:
```
python -m distributed.cli.dask_scheduler
```
Since the default values even cause the debug messages to be printed, there's no need to set the UCX config options to see the debug messages above.

@quasiben @pentschev 



Discussion:

Resolving patch:
diff --git a/distributed/comm/ucx.py b/distributed/comm/ucx.py
--- a/distributed/comm/ucx.py
+++ b/distributed/comm/ucx.py
@@ -512,11 +512,9 @@ def _scrub_ucx_config():
             options["NET_DEVICES"] = net_devices
 
     # ANY UCX options defined in config will overwrite high level dask.ucx flags
-    valid_ucx_keys = list(get_config().keys())
-    for k, v in dask.config.get("ucx").items():
-        if k in valid_ucx_keys:
-            options[k] = v
-        else:
+    valid_ucx_vars = list(get_config().keys())
+    for k, v in options.items():
+        if k not in valid_ucx_vars:
             logger.debug(
                 "Key: %s with value: %s not a valid UCX configuration option" % (k, v)
             )


Test patch:
Repository: dask/distributed

Issue:
Bokeh is validating properties
I'm noticing that my laptop is hot even with an idle cluster.  I brought up the profile-server chart to look at what was going on and it looks like Bokeh is pretty active validating properties, which it shouldn't be doing.  I looked at the line profile and found that for example, this line was triggering validation

https://github.com/dask/distributed/blob/5f01fe6010a0a3621956920d0260ac5f0111f17d/distributed/dashboard/components/scheduler.py#L2408-L2413

despite the containing method being wrapped with `@without_property_validation`

cc @jsignell @bryevdv 


Discussion:

Resolving patch:
diff --git a/distributed/dashboard/components/__init__.py b/distributed/dashboard/components/__init__.py
--- a/distributed/dashboard/components/__init__.py
+++ b/distributed/dashboard/components/__init__.py
@@ -79,6 +79,7 @@ def add_periodic_callback(doc, component, interval):
     _attach(doc, component)
 
 
+@without_property_validation
 def update(ref):
     comp = ref()
     if comp is not None:


Test patch:
Repository: dask/distributed

Issue:
Typo in docs relating to example code for worker plugins
**Issue**:
From plugins.rst, there's an example illustrating the use of workerplugins
```python
class ErrorLogger(WorkerPlugin):
    def __init__(self, logger):
        self.logger = logger
...
```
```python
plugin = ErrorLogger()
client.register_worker_plugin(plugin)
```
However the code incorrectly instantiates ErrorLogger since it's missing the actual logger

**Solution**
```python
import logger
plugin = ErrorLogger(logger)
```
This requires a minimal change and thus I intend on submitting an MR shortly. 


Discussion:
@crendoncoiled Thank you for reporting this, I can reproduce the error.  A PR will be welcomed, one small thing I think we should do:

```python
import logging
plugin = ErrorLogger(logging)
```

Resolving patch:
diff --git a/distributed/diagnostics/plugin.py b/distributed/diagnostics/plugin.py
--- a/distributed/diagnostics/plugin.py
+++ b/distributed/diagnostics/plugin.py
@@ -127,7 +127,8 @@ class WorkerPlugin:
     ...                 exc_info=exc_info
     ...             )
 
-    >>> plugin = ErrorLogger()
+    >>> import logging
+    >>> plugin = ErrorLogger(logging)
     >>> client.register_worker_plugin(plugin)  # doctest: +SKIP
     """
 


Test patch:
Repository: dask/distributed

Issue:
`Worker.log_event` is not threadsafe / `BatchedSend.send` not threadsafe
**What happened**:

Calling `get_worker().log_event(...)` from multiple worker threads can fail (or probably corrupt the BatchedSend), since `BatchedSend.send` is not thread-safe.

I saw a traceback like
```python-tracebak
/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/worker.py in log_event()

/opt/conda/envs/coiled/lib/python3.8/site-packages/distributed/batched.py in send()

/opt/conda/envs/coiled/lib/python3.8/site-packages/tornado/locks.py in set()

RuntimeError: Set changed size during iteration
```

This is probably coming from the `_background_send` coroutine calling `wait` on the tornado Event (adding to its `_waiters` set) while the `send` call in a worker thread calls `set` (iterating over the `_waiters` set).

**What you expected to happen**:

Users don't have to manage thread-safety of this API; it locks automatically.

**Anything else we need to know?**:

This is certainly a rarely-used API, but it is advertised in the docs: http://distributed.dask.org/en/stable/logging.html#structured-logs.

Easy to fix (`sync`?), though not a high priority, just wanted to note it for posterity.

**Environment**:

- Dask version: 2021.11.2
- Python version: 3.8.7
- Operating System: linux
- Install method (conda, pip, source): pip


Discussion:
I'm wondering what the worst case of a non-thread safe interaction would be. As the post points out, this can raise a RuntimeError whenever a send is called. That typically happens after a message has been queued up in the buffer so the worst case for a *single message* submit would be that a message submission is delayed until the next wake. No big deal

However, all patterns that send many messages subsequently would be at risk of dropping a message

```python
for msg in msgs:
    batched.send(msg)
```

I'm further wondering if this race condition could not corrupt the `tornado.Event` itself, such that the `BatchedSend._background_send` might deadlock since actions on futures (under the hood) are also not thread safe and might leave the critical future in a corrupt state

---

Today, we investigated this topic briefly around a spurious connection failure but I don't think any of this could actually corrupt the connection itself.

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -922,12 +922,13 @@ def logs(self):
         return self._deque_handler.deque
 
     def log_event(self, topic, msg):
-        self.batched_stream.send(
+        self.loop.add_callback(
+            self.batched_stream.send,
             {
                 "op": "log-event",
                 "topic": topic,
                 "msg": msg,
-            }
+            },
         )
 
     @property


Test patch:
Repository: dask/distributed

Issue:
TextProgressBar and logging/print
In [TextProgressBar](https://github.com/dask/distributed/blob/4081a0e09febd9099f04a1db9772f6cd95203b2e/distributed/diagnostics/progressbar.py#L138)

```python 
    def _draw_stop(self, **kwargs):
        sys.stdout.write("\r")
        sys.stdout.flush()
```

probably `\n` is better than `\r`? otherwise, other `stdout` messages from `logging` or `print` can't be displayed well.


Discussion:
Thanks for opening up an issue @kwunlyou. I believe the use of `\r` is to prevent outputting many lines from the progress bar (one line each time the progress bar is drawn). What's the current behavior when using the text progress bar and printing/logging something? I would be in favor of improving the current printing/logging behavior as long as we don't print a separate progress bar line each time the bar is drawn. 
@jrbourbeau 
>I believe the use of \r is to prevent outputting many lines from the progress bar (one line each time the progress bar is drawn). 

My concern is about the `\r` in `_draw_stop` but not in `_draw_bar`. I think it is better to use `\n` in `_draw_stop` otherwise this could mess up the following output of `stdout`, such as 

![image](https://user-images.githubusercontent.com/1352316/73248551-12de6800-41ab-11ea-8acc-d63d7fbc36f1.png)

in the figure, `d | 32.9s` is from the progress bar

the right output should be like 
![image](https://user-images.githubusercontent.com/1352316/73248841-926c3700-41ab-11ea-89bc-b31317731bea.png)





Thanks for the screenshots @kwunlyou, those are useful. At this point I think it may be easier to review a PR so things can be tested locally. Is that something you're interested in doing? (no obligation)
What's the status on this issue? Is it on your radar by any chance?
I don't think anyone has picked this up. Is this something you are interested in doing, Amin? 🙂
@jakirkham Sure, I just don't know where to start. Could you please give me some pointers?
I think this is mostly covered by other comments in the issue. Would suggest reading back over them and trying those changes in a new PR

Resolving patch:
diff --git a/distributed/diagnostics/progressbar.py b/distributed/diagnostics/progressbar.py
--- a/distributed/diagnostics/progressbar.py
+++ b/distributed/diagnostics/progressbar.py
@@ -163,7 +163,7 @@ def _draw_bar(self, remaining, all, **kwargs):
             sys.stdout.flush()
 
     def _draw_stop(self, **kwargs):
-        sys.stdout.write("\r")
+        sys.stdout.write("\33[2K\r")
         sys.stdout.flush()
 
 


Test patch:
Repository: dask/distributed

Issue:
the "distributed.worker" logger gains ~thousand DequeHandler instances during a pytest run
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

running `len(logging.getLogger("distributed.worker").handlers)` shows a few thousand `DequeHandler` instances in there.
This means that logging messages from the worker or adding handlers to the worker gradually gets slower and slower as the test suite progresses.

the handlers are supposed to be cleared when the worker is GC'd by
https://github.com/dask/distributed/blob/1da51994cc84c0897589a4476cd24073147054b6/distributed/node.py#L88 however https://github.com/dask/distributed/blob/1da51994cc84c0897589a4476cd24073147054b6/distributed/utils.py#L1131-L1142 keeps a permanent strong reference to the worker:

![image](https://user-images.githubusercontent.com/413772/159285688-6305720a-ee4c-4bef-a375-09a6c07fc36b.png)

see https://github.com/dask/distributed/pull/4481 and https://github.com/dask/distributed/issues/4469



Discussion:
the reference chain is from `objgraph.find_backref_chain` https://mg.pov.lt/objgraph/#memory-leak-example
If the performance problem is an actual concern, ensuring all handlers are async is probably the most straight forward and would resolve a few other issues around exception handling

some earlier and related work
- https://github.com/dask/distributed/pull/5443
- https://github.com/dask/distributed/pull/4734
- https://github.com/dask/distributed/issues/5483

Resolving patch:
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -1124,21 +1124,8 @@ def color_of(x, palette=palette):
     return palette[n % len(palette)]
 
 
-def _iscoroutinefunction(f):
-    return inspect.iscoroutinefunction(f) or gen.is_coroutine_function(f)
-
-
-@functools.lru_cache(None)
-def _iscoroutinefunction_cached(f):
-    return _iscoroutinefunction(f)
-
-
 def iscoroutinefunction(f):
-    # Attempt to use lru_cache version and fall back to non-cached version if needed
-    try:
-        return _iscoroutinefunction_cached(f)
-    except TypeError:  # unhashable type
-        return _iscoroutinefunction(f)
+    return inspect.iscoroutinefunction(f) or gen.is_coroutine_function(f)
 
 
 @contextmanager


Test patch: