Repository: dask/dask

Issue:
'remove stringification' causes dask/dask CI to hang
CI broke with the version bump to `2023.8.1`. For some reason, some jobs started taking much longer and time out.

Commit: https://github.com/dask/dask/commit/5f456cc899890d11da0bf528412858e19665eccd
Workflow logs: https://github.com/dask/dask/actions/runs/5906801262



Discussion:
Thanks @hendrikmakait. The failures on that commit are from https://github.com/dask/dask/issues/10455, which I believe you've since fixed
@jrbourbeau: Sorry, mixed things up. Those failures are fixed, the new ones originate from https://github.com/dask/dask/commit/709e977b7ca9cdd1941c495056d0f89c93600b7e

CI: https://github.com/dask/dask/actions/runs/5965469810/job/16182976640
cc @crusaderky, though I'm not sure if the CI problems are caused by the PR.
Ah, I see. I'm also not sure if the changes in that PR are cause CI issues, or if it's something else like another library releasing. Seeing if reverting helps in https://github.com/dask/dask/pull/10469
Thanks for checking, I also don't see how the changes could cause issues given that the only failing change was a minor adjustment to the tests.
They're not taking much longer - it looks like they're hanging on 99%. I can't see how #10456 can cause the issue...
Reverting both #10456 and dask/distributed#8127 has no effect.

mamba env export doesn't highlight anything at first glance, but at the moment I don't have a better lead:
```
@@ -15 +15 @@
- -  argon2-cffi=21.3.0=pyhd8ed1ab_0
+ -  argon2-cffi=23.1.0=pyhd8ed1ab_0
@@ -50 +50 @@
- -  c-blosc2=2.10.0=hb4ffafa_0
+ -  c-blosc2=2.10.2=hb4ffafa_0
@@ -84 +83,0 @@
- -  flit-core=3.9.0=pyhd8ed1ab_0
@@ -115 +114 @@
- -  identify=2.5.26=pyhd8ed1ab_0
+ -  identify=2.5.27=pyhd8ed1ab_0
@@ -141 +140 @@
- -  kiwisolver=1.4.4=py39hf939315_1
+ -  kiwisolver=1.4.5=py39h7633fee_0
@@ -190 +189 @@
- -  libprotobuf=3.21.12=h3eb15da_0
+ -  libprotobuf=3.21.12=hfc55251_1
@@ -238 +237 @@
- -  nss=3.89=he45b914_0
+ -  nss=3.92=h1d7d5a4_0
@@ -337,2 +336,2 @@
- -  websocket-client=1.6.1=pyhd8ed1ab_0
- -  wheel=0.41.1=pyhd8ed1ab_0
+ -  websocket-client=1.6.2=pyhd8ed1ab_0
+ -  wheel=0.41.2=pyhd8ed1ab_0
@@ -372,2 +371,2 @@
- -  dask==2023.8.1
- -  distributed==2023.8.1
+ -  dask==2023.8.1+1.g709e977b7
+ -  distributed==2023.8.1+5.g22eb33a2
```
The hang is introduced somewhere between distributed 2023.8.1 (works) and dask/distributed#8083 (hangs). 
Both are before https://github.com/dask/dask/pull/10456 and https://github.com/dask/distributed/pull/8127.

Bisecting...
Bisection done.
dask/dask git tip is:

1. fully green vs. https://github.com/dask/distributed/pull/8125
2. hanging on the next PR, https://github.com/dask/distributed/pull/8083
I'm a bit surprised that `mindeps-optional` hangs, while all other `mindeps-*` environments work.
This means that the test that causes the whole interpreter to hang runs only if one of these dependencies is installed:
```
- bokeh=2.4.2
- pyarrow=7.0
- lz4=4.3.2
- cachey=0.1.1
- crick=0.0.3
- cytoolz=0.11.0
- dask-ml=1.4.0
- fastavro=1.1.0
- fastparquet=0.8.2
- h5py=2.10.0
- ipycytoscape=1.0.1
- IPython=7.16.1
- matplotlib=3.4.1
- mimesis=5.3.0
- mmh3=2.5.1
- python-cityhash=0.4.6
- python-graphviz=0.8.4
- python-snappy=0.5.4
- python-xxhash=2.0.0
- s3fs=2021.9.0
- scikit-image=0.17.2
- scipy=1.5.2
- sparse=0.12.0
- sqlalchemy=1.4.16
- tiledb-py=0.8.1
- zarr=2.12.0
```

https://github.com/dask/distributed/pull/8083 broke #9072.
The lack of pytest-timeout caused the whole test suite to hang instead of a single test to fail.

#10482 adds pytest-timeout.

Currently investigating now if the test for #9072, `test_blockwise_cull_allows_numpy_dtype_keys`, is still relevant to reproducing the issue - in other words, if https://github.com/dask/distributed/pull/8083 actually breaks compatibility with dask-geopandas - or if it's just a test issue.

Resolving patch:
diff --git a/dask/blockwise.py b/dask/blockwise.py
--- a/dask/blockwise.py
+++ b/dask/blockwise.py
@@ -587,7 +587,7 @@ def cull(
         output_blocks: set[tuple[int, ...]] = set()
         for key in keys:
             if key[0] == self.output:
-                output_blocks.add(tuple(map(int, key[1:])))
+                output_blocks.add(key[1:])
         culled_deps = self._cull_dependencies(all_hlg_keys, output_blocks)
         out_size_iter = (self.dims[i] for i in self.output_indices)
         if prod(out_size_iter) != len(culled_deps):


Test patch:
diff --git a/dask/tests/test_distributed.py b/dask/tests/test_distributed.py
--- a/dask/tests/test_distributed.py
+++ b/dask/tests/test_distributed.py
@@ -30,7 +30,6 @@
 from dask.blockwise import Blockwise
 from dask.delayed import Delayed
 from dask.distributed import futures_of, wait
-from dask.highlevelgraph import HighLevelGraph
 from dask.layers import ShuffleLayer, SimpleShuffleLayer
 from dask.utils import get_named_args, get_scheduler_lock, tmpdir, tmpfile
 from dask.utils_test import inc
@@ -585,28 +584,6 @@ def test_blockwise_different_optimization(c):
     np.testing.assert_equal(y_value, expected)
 
 
-def test_blockwise_cull_allows_numpy_dtype_keys(c):
-    # Regression test for https://github.com/dask/dask/issues/9072
-    da = pytest.importorskip("dask.array")
-    np = pytest.importorskip("numpy")
-
-    # Create a multi-block array.
-    x = da.ones((100, 100), chunks=(10, 10))
-
-    # Make a layer that pulls a block out of the array, but
-    # refers to that block using a numpy.int64 for the key rather
-    # than a python int.
-    name = next(iter(x.dask.layers))
-    block = {("block", 0, 0): (name, np.int64(0), np.int64(1))}
-    dsk = HighLevelGraph.from_collections("block", block, [x])
-    arr = da.Array(dsk, "block", ((10,), (10,)), dtype=x.dtype)
-
-    # Stick with high-level optimizations to force serialization of
-    # the blockwise layer.
-    with dask.config.set({"optimization.fuse.active": False}):
-        da.assert_eq(np.ones((10, 10)), arr, scheduler=c)
-
-
 @gen_cluster(client=True)
 async def test_combo_of_layer_types(c, s, a, b):
     """Check pack/unpack of a HLG that has every type of Layers!"""