internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/distributed

Issue:
Cluster.name (and other _cluster_info items) get carried over to subsequent Clusters
( if `Cluster.__init__` is not called before `Cluster.name.__set__` )


```python
import sys

from distributed.deploy import Cluster

class ExampleCluster(Cluster):
    def __init__(self, name, *args, **kwargs):
        self.name = f"example-cluster-{name}"
        super().__init__(*args, **kwargs)


def main():

    print(
        Cluster()
    )  # Cluster(e174b1b2, '<Not Connected>', workers=0, threads=0, memory=0 B)
    print(
        ExampleCluster("demo")
    )  # ExampleCluster(example-cluster-demo, '<Not Connected>', workers=0, threads=0, memory=0 B)
    print(
        Cluster()
    )  # Cluster(example-cluster-demo, '<Not Connected>', workers=0, threads=0, memory=0 B)
    return 0


if __name__ == "__main__":
    sys.exit(main())
```

see also https://github.com/dask/dask-kubernetes/issues/516
see also https://github.com/dask/distributed/pull/5305#discussion_r886822632


Discussion:
@jacobtomlinson 

Resolving patch:
diff --git a/distributed/deploy/cluster.py b/distributed/deploy/cluster.py
--- a/distributed/deploy/cluster.py
+++ b/distributed/deploy/cluster.py
@@ -55,7 +55,6 @@ class Cluster(SyncMethodMixin):
     """
 
     _supports_scaling = True
-    _cluster_info: dict = {}
 
     def __init__(
         self,
@@ -84,11 +83,9 @@ def __init__(
         if name is None:
             name = str(uuid.uuid4())[:8]
 
-        # Mask class attribute with instance attribute
         self._cluster_info = {
             "name": name,
             "type": typename(type(self)),
-            **type(self)._cluster_info,
         }
         self.status = Status.created
 


Test patch:
diff --git a/distributed/deploy/tests/test_cluster.py b/distributed/deploy/tests/test_cluster.py
--- a/distributed/deploy/tests/test_cluster.py
+++ b/distributed/deploy/tests/test_cluster.py
@@ -35,16 +35,3 @@ async def test_logs_deprecated():
     async with Cluster(asynchronous=True) as cluster:
         with pytest.warns(FutureWarning, match="get_logs"):
             cluster.logs()
-
-
-@gen_test()
-async def test_cluster_info(loop_in_thread):
-    class FooCluster(Cluster):
-        def __init__(self):
-            self._cluster_info["foo"] = "bar"
-            super().__init__(asynchronous=False, loop=loop_in_thread)
-
-    cluster = FooCluster()
-    assert "foo" in cluster._cluster_info  # exists before start() called
-    with cluster:  # start and stop the cluster to avoid a resource warning
-        pass
Repository: dask/distributed

Issue:
API change in #6363 broke dask gateway
In #6363, a parameter was removed that makes distributed in compatible with Dask Gateway: https://github.com/dask/dask-gateway/pull/573

I'd recommend we restore that parameter and deprecate it. If it's impossible to restore the previous functionality, we could restore the parameter and perhaps ignore it, just for the signature compatibility (I don't know if dask-gateway was relying on the previous behavior).


Discussion:
Looking at the linked `dask-gateway` PR, it looks like removing the `fast=` keyword from `Scheduler.close` is the parameter you're referring to @TomAugspurger. cc @fjetter @gjoseph92 who were actively involved on https://github.com/dask/distributed/pull/6363
Question, are we doing upstream testing in dask-gateway?  If not, does anyone want to take this on?  (@TomAugspurger ?)

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -3354,13 +3354,18 @@ def del_scheduler_file():
         setproctitle(f"dask-scheduler [{self.address}]")
         return self
 
-    async def close(self):
+    async def close(self, fast=None, close_workers=None):
         """Send cleanup signal to all coroutines then wait until finished
 
         See Also
         --------
         Scheduler.cleanup
         """
+        if fast is not None or close_workers is not None:
+            warnings.warn(
+                "The 'fast' and 'close_workers' parameters in Scheduler.close have no effect and will be removed in a future version of distributed.",
+                FutureWarning,
+            )
         if self.status in (Status.closing, Status.closed):
             await self.finished()
             return


Test patch:
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -3595,3 +3595,9 @@ async def test_worker_state_unique_regardless_of_address(s, w):
     assert ws1 is not ws2
     assert ws1 != ws2
     assert hash(ws1) != ws2
+
+
+@gen_cluster(nthreads=[("", 1)])
+async def test_scheduler_close_fast_deprecated(s, w):
+    with pytest.warns(FutureWarning):
+        await s.close(fast=True)
Repository: dask/distributed

Issue:
TypeError: not enough arguments for format string
In https://github.com/dask/distributed/pull/6566, the following line was added/moved:

https://github.com/dask/distributed/blob/8aab5e840aa48e9574d286c5119f023a426077dd/distributed/worker_state_machine.py#L1195

We're missing a stimulus_id argument to the format string, which results in a lot of spammy errors from logging:

```
2022-06-16T16:42:29.486762121Z --- Logging error ---
2022-06-16T16:42:29.486764931Z Traceback (most recent call last):
2022-06-16T16:42:29.486767752Z   File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit
2022-06-16T16:42:29.486770532Z     msg = self.format(record)
2022-06-16T16:42:29.486773632Z   File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format
2022-06-16T16:42:29.486779212Z     return fmt.format(record)
2022-06-16T16:42:29.486782252Z   File "/usr/local/lib/python3.9/logging/__init__.py", line 663, in format
2022-06-16T16:42:29.486785202Z     record.message = record.getMessage()
2022-06-16T16:42:29.486788192Z   File "/usr/local/lib/python3.9/logging/__init__.py", line 367, in getMessage
2022-06-16T16:42:29.486791072Z     msg = msg % self.args
2022-06-16T16:42:29.486793722Z TypeError: not enough arguments for format string
```


Discussion:
cc @crusaderky as you made the change. I can make a PR to remove the `stimulus_id=%s` section

Resolving patch:
diff --git a/distributed/worker_state_machine.py b/distributed/worker_state_machine.py
--- a/distributed/worker_state_machine.py
+++ b/distributed/worker_state_machine.py
@@ -1191,8 +1191,8 @@ def _purge_state(self, ts: TaskState) -> None:
         who_has)
         This is idempotent
         """
+        logger.debug("Purge task: %s", ts)
         key = ts.key
-        logger.debug("Purge task key: %s state: %s; stimulus_id=%s", ts.key, ts.state)
         self.data.pop(key, None)
         self.actors.pop(key, None)
 


Test patch:
Repository: dask/distributed

Issue:
Fix profile on msgpack 1.0.6/Windows
- Closes #8212
- Iterates on #3455
- Twin of #8214

Make sure that profile doesn't reach the point where sizeof() starts raising (at depth 140).
This is problematic with msgpack 1.0.6 on Windows because its max recursion depth fell from 512 to 256.

The remaining exception in the unit tests is fixed by #8214


Discussion:

Resolving patch:
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -30,6 +30,19 @@ class LocalCluster(SpecCluster):
     ----------
     n_workers: int
         Number of workers to start
+    memory_limit: str, float, int, or None, default "auto"
+        Sets the memory limit *per worker*.
+
+        Notes regarding argument data type:
+
+        * If None or 0, no limit is applied.
+        * If "auto", the total system memory is split evenly between the workers.
+        * If a float, that fraction of the system memory is used *per worker*.
+        * If a string giving a number of bytes (like ``"1GiB"``), that amount is used *per worker*.
+        * If an int, that number of bytes is used *per worker*.
+
+        Note that the limit will only be enforced when ``processes=True``, and the limit is only
+        enforced on a best-effort basis — it's still possible for workers to exceed this limit.
     processes: bool
         Whether to use processes (True) or threads (False).  Defaults to True, unless
         worker_class=Worker, in which case it defaults to False.


Test patch:
Repository: dask/distributed

Issue:
Importing from distributed shows pyright error
**What happened**:
When type-checking a program that imports from distributed with pyright, an error is accused:

```python
# foo.py
from distributed import Client
print(Client)
```

```
pyright foo.py
...
  /tmp/test-area/foo.py:1:25 - error: "Client" is not exported from module "distributed"
    Import from "distributed.client" instead (reportPrivateImportUsage)
1 error, 0 warnings, 0 informations
```

**What you expected to happen**:
I expect the package to be correctly typed, following PEP 484


**Anything else we need to know?**:

PEP 484 states that

> Modules and variables imported into the stub are not considered exported from the stub unless the import uses the `import ... as ... form` or the equivalent `from ... import ... as ... form`

and Pyright follows this guideline, although mypy doesn't.

**Environment**:

- Dask version: 2022.8.0
- Python version: 3.10.5
- Operating System: Arch linux
- Install method (conda, pip, source): pip inside an environment


Discussion:

Resolving patch:
diff --git a/distributed/__init__.py b/distributed/__init__.py
--- a/distributed/__init__.py
+++ b/distributed/__init__.py
@@ -99,3 +99,65 @@ def _():
     """
     global _python_shutting_down
     _python_shutting_down = True
+
+
+__all__ = [
+    "Actor",
+    "ActorFuture",
+    "Adaptive",
+    "BaseActorFuture",
+    "CancelledError",
+    "Client",
+    "CompatibleExecutor",
+    "Environ",
+    "Event",
+    "Future",
+    "KilledWorker",
+    "LocalCluster",
+    "Lock",
+    "MultiLock",
+    "Nanny",
+    "NannyPlugin",
+    "PipInstall",
+    "Pub",
+    "Queue",
+    "Reschedule",
+    "SSHCluster",
+    "Scheduler",
+    "SchedulerPlugin",
+    "Security",
+    "Semaphore",
+    "SpecCluster",
+    "Status",
+    "Sub",
+    "TimeoutError",
+    "UploadDirectory",
+    "UploadFile",
+    "Variable",
+    "Worker",
+    "WorkerPlugin",
+    "as_completed",
+    "config",
+    "connect",
+    "dask",
+    "default_client",
+    "fire_and_forget",
+    "futures_of",
+    "get_client",
+    "get_task_metadata",
+    "get_task_stream",
+    "get_versions",
+    "get_worker",
+    "local_client",
+    "performance_report",
+    "print",
+    "progress",
+    "rejoin",
+    "rpc",
+    "secede",
+    "sync",
+    "wait",
+    "warn",
+    "widgets",
+    "worker_client",
+]


Test patch:
Repository: dask/distributed

Issue:
Validate all state-based collections in `WorkerState`
In `validate_state`, we should ensure that all state-based collections only contain those tasks that are in the desired state. We already do this to some extent with https://github.com/dask/distributed/blob/main/distributed/worker_state_machine.py#L3180-L3188, but we should extend this to include validation for the other collections as well, notably `self.constrained` and `self.ready`.

XREF: https://github.com/dask/distributed/pull/6698#pullrequestreview-1034032402


Discussion:
Blocked by #6699/#6716

Resolving patch:
diff --git a/distributed/worker_state_machine.py b/distributed/worker_state_machine.py
--- a/distributed/worker_state_machine.py
+++ b/distributed/worker_state_machine.py
@@ -3262,11 +3262,6 @@ def validate_state(self) -> None:
             assert ts.state == "ready"
         for ts in self.constrained:
             assert ts.state == "constrained"
-        # FIXME https://github.com/dask/distributed/issues/6708
-        # for ts in self.in_flight_tasks:
-        #     assert ts.state == "flight" or (
-        #         ts.state in ("cancelled", "resumed") and ts.previous == "flight"
-        #     )
         for ts in self.executing:
             assert ts.state == "executing" or (
                 ts.state in ("cancelled", "resumed") and ts.previous == "executing"


Test patch:
Repository: dask/distributed

Issue:
Deep copy causes memory flare on unspill
- Twin of https://github.com/dask/distributed/issues/5107

When spilling a key with pickle5 buffers (e.g. numpy or pandas), you'll get n+2 frames, with the first two being the header and non-buffer pickle output.
This data is written contiguously to a file.

When unspilling the key, the pickle5 buffers are deep-copied by deserialization; this is both time-consuming (https://github.com/dask/distributed/issues/7351#issuecomment-1349939552) and causes a memory flare.

Note that this issue only affects uncompressible data (or when lz4 has not been installed); when the buffers are compressed, there is an unavoidable moment where you have both the compressed and uncompressed copy in memory.


Discussion:

Resolving patch:
diff --git a/distributed/protocol/numpy.py b/distributed/protocol/numpy.py
--- a/distributed/protocol/numpy.py
+++ b/distributed/protocol/numpy.py
@@ -134,7 +134,12 @@ def deserialize_numpy_ndarray(header, frames):
     x = np.ndarray(shape, dtype=dt, buffer=frame, strides=header["strides"])
     if not writeable:
         x.flags.writeable = False
-    else:
+    elif not x.flags.writeable:
+        # This should exclusively happen when the underlying buffer is read-only, e.g.
+        # a read-only mmap.mmap or a bytes object.
+        # Specifically, these are the known use cases:
+        # 1. decompressed output of a buffer that was not sharded
+        # 2. unspill with zict <2.3.0 (https://github.com/dask/zict/pull/74)
         x = np.require(x, requirements=["W"])
 
     return x
diff --git a/distributed/spill.py b/distributed/spill.py
--- a/distributed/spill.py
+++ b/distributed/spill.py
@@ -18,6 +18,7 @@
 logger = logging.getLogger(__name__)
 has_zict_210 = parse_version(zict.__version__) >= parse_version("2.1.0")
 has_zict_220 = parse_version(zict.__version__) >= parse_version("2.2.0")
+has_zict_230 = parse_version(zict.__version__) >= parse_version("2.3.0")
 
 
 class SpilledSize(NamedTuple):
@@ -381,7 +382,7 @@ def __init__(self, spill_directory: str, max_weight: int | Literal[False] = Fals
     def __getitem__(self, key: str) -> Any:
         t0 = perf_counter()
         pickled = self.d[key]
-        assert isinstance(pickled, bytes)
+        assert isinstance(pickled, bytearray if has_zict_230 else bytes)
         t1 = perf_counter()
         out = self.load(pickled)  # type: ignore
         t2 = perf_counter()


Test patch:
Repository: dask/distributed

Issue:
List full dashboard address in scheduler logs
When users run `dask scheduler`, logs like the following are emitted

```
2023-02-14 10:17:38,016 - distributed.scheduler - INFO - -----------------------------------------------
2023-02-14 10:17:39,076 - distributed.scheduler - INFO - State start
2023-02-14 10:17:39,080 - distributed.scheduler - INFO - -----------------------------------------------
2023-02-14 10:17:39,082 - distributed.scheduler - INFO -   Scheduler at:  tcp://<ip-address>:8786
2023-02-14 10:17:39,082 - distributed.scheduler - INFO -   dashboard at:                     :8787
```

It'd be nice if instead of `dashboard at:                     :8787`, we listed the full dashboard address. That makes it a bit easier for users to navigate to the dashboard. 


Discussion:

Resolving patch:
diff --git a/distributed/node.py b/distributed/node.py
--- a/distributed/node.py
+++ b/distributed/node.py
@@ -171,7 +171,7 @@ def start_http_server(
         bound_addresses = get_tcp_server_addresses(self.http_server)
 
         # If more than one address is configured we just use the first here
-        self.http_server.port = bound_addresses[0][1]
+        self.http_server.address, self.http_server.port = bound_addresses[0]
         self.services["dashboard"] = self.http_server
 
         # Warn on port changes
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -101,6 +101,7 @@
     All,
     TimeoutError,
     empty_context,
+    format_dashboard_link,
     get_fileno_limit,
     key_split_group,
     log_errors,
@@ -3860,8 +3861,13 @@ async def start_unsafe(self):
 
         for listener in self.listeners:
             logger.info("  Scheduler at: %25s", listener.contact_address)
-        for k, v in self.services.items():
-            logger.info("%11s at: %25s", k, "%s:%d" % (listen_ip, v.port))
+        for name, server in self.services.items():
+            if name == "dashboard":
+                addr = get_address_host(listener.contact_address)
+                link = format_dashboard_link(addr, server.port)
+            else:
+                link = f"{listen_ip}:{server.port}"
+            logger.info("%11s at:  %25s", name, link)
 
         if self.scheduler_file:
             with open(self.scheduler_file, "w") as f:


Test patch:
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -2870,6 +2870,7 @@ async def test_multiple_listeners():
     log = log.getvalue()
     assert re.search(r"Scheduler at:\s*tcp://", log)
     assert re.search(r"Scheduler at:\s*inproc://", log)
+    assert re.search(r"dashboard at:\s*http://", log)
 
 
 @gen_cluster(nthreads=[("127.0.0.1", 1)])
Repository: dask/distributed

Issue:
Add Prometheus counter for `SystemMonitor.last_time` to improve GIL contention metric
Currently, the loose coupling between the system monitor's update interval and the Prometheus scraping interval can cause artifacts like a relative GIL contention > 1 (https://github.com/dask/distributed/pull/7651#issuecomment-1490571845). By exposing the system monitor's update timestamp as a Counter, we would have a synchronized timestamp available in Prometheus to serve as the basis for rate calculations. This should make such artifacts impossible.

cc @ntabris, @gjoseph92, @milesgranger: Thoughts?


Discussion:
I think that we should then also convert the GIL contention metric to the (estimated) total GIL contention time. This would make it easier to understand the metric value than a cumulative sum of percentages and any difference between to measurements would be the same as for the time metric (i.e., an absolute timedelta).
Makes sense to me.

Resolving patch:
diff --git a/distributed/http/scheduler/prometheus/core.py b/distributed/http/scheduler/prometheus/core.py
--- a/distributed/http/scheduler/prometheus/core.py
+++ b/distributed/http/scheduler/prometheus/core.py
@@ -59,6 +59,12 @@ def collect(self) -> Iterator[GaugeMetricFamily | CounterMetricFamily]:
                 value=self.server.monitor._cumulative_gil_contention,
             )
 
+        yield CounterMetricFamily(
+            self.build_name("last_time"),
+            "SystemMonitor last time",
+            value=self.server.monitor.last_time,
+        )
+
         tasks = GaugeMetricFamily(
             self.build_name("tasks"),
             "Number of tasks known by scheduler",


Test patch:
diff --git a/distributed/http/scheduler/tests/test_scheduler_http.py b/distributed/http/scheduler/tests/test_scheduler_http.py
--- a/distributed/http/scheduler/tests/test_scheduler_http.py
+++ b/distributed/http/scheduler/tests/test_scheduler_http.py
@@ -108,6 +108,7 @@ async def test_prometheus(c, s, a, b):
         "dask_scheduler_clients",
         "dask_scheduler_desired_workers",
         "dask_scheduler_workers",
+        "dask_scheduler_last_time",
         "dask_scheduler_tasks",
         "dask_scheduler_tasks_suspicious",
         "dask_scheduler_tasks_forgotten",
Repository: dask/distributed

Issue:
shuffle_transfer failed during shuffle in `test_fused_blockwise_dataframe_merge`
**Describe the issue**:

Distributed test started failing in dask/dask CI with distributed `main`.

The error looks like this:

```
FAILED dask/tests/test_distributed.py::test_fused_blockwise_dataframe_merge[True] - RuntimeError: shuffle_barrier failed during shuffle 1d8d714896c0fa093a11a0ddfed35635
FAILED dask/tests/test_distributed.py::test_fused_blockwise_dataframe_merge[False] - RuntimeError: shuffle_transfer failed during shuffle 1d8d714896c0fa093a11a0ddfed35635
```

Example workflow:

https://github.com/dask/dask/actions/runs/5555724247/jobs/10147277695

**Anything else we need to know?**:

**Environment**:

- Dask version: 2023.7.0
- Python version:
- Operating System:
- Install method (conda, pip, source):



Discussion:
cc @hendrikmakait @fjetter @crusaderky 
Thanks for flagging this, @j-bennet. I can reproduce this locally when installing `pyarrow 13.0.0.dev517`. If using `pyarrow=12`, the test succeeds.
I've created a dedicated issue for this: https://github.com/dask/distributed/issues/8007

Resolving patch:
diff --git a/distributed/shuffle/_worker_plugin.py b/distributed/shuffle/_worker_plugin.py
--- a/distributed/shuffle/_worker_plugin.py
+++ b/distributed/shuffle/_worker_plugin.py
@@ -951,7 +951,7 @@ def split_by_worker(
     # bytestream such that it cannot be deserialized anymore
     t = pa.Table.from_pandas(df, preserve_index=True)
     t = t.sort_by("_worker")
-    codes = np.asarray(t.select(["_worker"]))[0]
+    codes = np.asarray(t["_worker"])
     t = t.drop(["_worker"])
     del df
 
@@ -983,7 +983,7 @@ def split_by_partition(t: pa.Table, column: str) -> dict[Any, pa.Table]:
     partitions.sort()
     t = t.sort_by(column)
 
-    partition = np.asarray(t.select([column]))[0]
+    partition = np.asarray(t[column])
     splits = np.where(partition[1:] != partition[:-1])[0] + 1
     splits = np.concatenate([[0], splits])
 


Test patch:
Repository: dask/distributed

Issue:
LocalCluster: discrepancy default option code & documentation
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**Describe the issue**:
The default scheduler port of a LocalCluster is 0 (random port), not 8786 as mentioned in [the documentation](https://distributed.dask.org/en/stable/api.html#distributed.LocalCluster).



**Minimal Complete Verifiable Example**:

```python
# Put your MCVE code here
```

**Anything else we need to know?**:

**Environment**:

- Dask version: dask-2023.7.1 distributed-2023.7.1
- Python version: 3.11.0
- Operating System:
- Install method (conda, pip, source):



Discussion:
Hi, thx for the report. Adjusted the docs sounds good. Are you interested in submitting a PR?

Resolving patch:
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -49,7 +49,7 @@ class LocalCluster(SpecCluster):
     threads_per_worker: int
         Number of threads per each worker
     scheduler_port: int
-        Port of the scheduler.  8786 by default, use 0 to choose a random port
+        Port of the scheduler. Use 0 to choose a random port (default). 8786 is a common choice.
     silence_logs: logging level
         Level of logs to print out to stdout.  ``logging.WARN`` by default.
         Use a falsey value like False or None for no change.


Test patch: