Repository: dask/dask
Pull request #11317 (2024-08-14 20:49:55)

Rename chunksize-tolerance option
- [ ] Closes #xxxx
- [ ] Tests added / passed
- [ ] Passes `pre-commit run --all-files`

@dcherian rightfully pointed out that this option is applicable in other areas as well, so removing the shuffle part. I think we should be able to get away with this since it's only been one release?

github-actions[bot]: ## Unit Test Results
_See [test report](https://dask.github.io/dask/test_report.html) for an extended history of previous test failures. This is useful for diagnosing flaky tests._

     13 files  ±0       13 suites  ±0   3h 2m 3s :stopwatch: + 1m 59s
 13 151 tests ±0   12 083 :white_check_mark: ±0   1 068 :zzz: ±0  0 :x: ±0 
137 048 runs  ±0  117 996 :white_check_mark: +2  19 052 :zzz:  - 2  0 :x: ±0 

Results for commit 69253920. ± Comparison against base commit 18239232.

[test-results]:data:application/gzip;base64,H4sIAP8fvWYC/02PSw7DIAwFr1Kx7sImQHAvUyUukVDzqQisqt695EeynHnySP6KzvduFo8bVvebmJOPJ71SaKKfxoWB5KLyGrcdNR78nBPzIiXY6pRv/1kvjS2ua3yfHRThQpjCbkIat3QNyu5c0lgTmSL3NIGWh7ukV76WeRoGHzMIQ1JXJIG0bYHQoWs4/6a4o9YBKlYkDdcofn9S4y6vGgEAAA==

phofl: We are doing our best to preserve input chunk sizes, sometimes we can't and this option configures how much we can deviate from the input chunk size upwards.
jrbourbeau: Sorry, I meant what happened with the config option name itself. It looks like this is a new config option that was added in the last release, xref https://github.com/dask/dask/pull/11262 -- I wasn't sure if this was an existing config that got renamed and we're now undoing that rename.

I'm fine either with a deprecation or not given there won't be much "existing code" that uses this option. For future reference, adding config deprecations just means adding an entry to this dictionary 

https://github.com/dask/dask/blob/058f1b69ee3115d63046acf60e537d358462a466/dask/config.py#L703-L724

So it's relatively lightweight 
phofl: Ah then I'll add the deprecation as well
phofl: It's a new option that is now being renamed, it didn't#t exist before the latest release

Patch:
diff --git a/dask/array/_shuffle.py b/dask/array/_shuffle.py
index fa1c6ccd6..78d2faa42 100644
--- a/dask/array/_shuffle.py
+++ b/dask/array/_shuffle.py
@@ -58,7 +58,7 @@ def shuffle(x, indexer: list[list[int]], axis: int, chunks: Literal["auto"] = "a
     the number of chunks small.
 
     The tolerance of increasing the chunk size is controlled by the configuration
-    "array.shuffle.chunksize-tolerance". The default value is 1.25.
+    "array.chunk-size-tolerance". The default value is 1.25.
 
     >>> y.chunks
     ((2,), (5, 3))
@@ -94,7 +94,7 @@ def _rechunk_other_dimensions(
     x: Array, longest_group: int, axis: int, chunks: Literal["auto"]
 ) -> Array:
     assert chunks == "auto", "Only auto is supported for now"
-    chunksize_tolerance = config.get("array.shuffle.chunksize-tolerance")
+    chunksize_tolerance = config.get("array.chunk-size-tolerance")
 
     if longest_group <= max(x.chunks[axis]) * chunksize_tolerance:
         # We are staying below our threshold, so don't rechunk
@@ -167,7 +167,7 @@ def _shuffle(chunks, indexer, axis, in_name, out_name, token):
 
     indexer = copy.deepcopy(indexer)
 
-    chunksize_tolerance = config.get("array.shuffle.chunksize-tolerance")
+    chunksize_tolerance = config.get("array.chunk-size-tolerance")
     chunk_size_limit = int(sum(chunks[axis]) / len(chunks[axis]) * chunksize_tolerance)
 
     # Figure out how many groups we can put into one chunk
diff --git a/dask/config.py b/dask/config.py
index 1376ae50f..ecb0f37d0 100644
--- a/dask/config.py
+++ b/dask/config.py
@@ -721,6 +721,7 @@ deprecations: dict[str, str | None] = {
     "dataframe.shuffle.algorithm": "dataframe.shuffle.method",
     "dataframe.shuffle-compression": "dataframe.shuffle.compression",
     "admin.traceback.shorten.what": "admin.traceback.shorten",  # changed in 2023.9.0
+    "array.shuffle.chunksize-tolerance": "array.chunk-size-tolerance",
 }
 
 
diff --git a/dask/dask-schema.yaml b/dask/dask-schema.yaml
index e05aac038..548c8e964 100644
--- a/dask/dask-schema.yaml
+++ b/dask/dask-schema.yaml
@@ -119,6 +119,13 @@ properties:
         description: |
           The default chunk size to target. Default is "128MiB".
 
+      chunk-size-tolerance:
+        type: number
+        description: |
+          Upper tolerance for different algorithms when creating output chunks.
+          Default is 1.25. This means that the algorithms can exceed
+          the average input chunk size along this dimension by 25%.
+
       rechunk:
         type: object
         properties:
@@ -135,17 +142,6 @@ properties:
               The graph growth factor above which task-based shuffling introduces
               an intermediate step.
 
-      shuffle:
-        type: object
-        properties:
-
-          chunksize-tolerance:
-            type: number
-            description: |
-              Upper tolerance for the shuffle algorithm when creating output chunks.
-              Default is 1.25. This means that the shuffle algorithm can exceed
-              the average input chunk size along this dimension by 25%.
-
       svg:
         type: object
         properties:
diff --git a/dask/dask.yaml b/dask/dask.yaml
index 6e35c10e4..e8d00aa14 100644
--- a/dask/dask.yaml
+++ b/dask/dask.yaml
@@ -21,11 +21,10 @@ dataframe:
 array:
   backend: "numpy"  # Backend array library for input IO and data creation
   chunk-size: "128MiB"
+  chunk-size-tolerance: 1.25  # Tolerance for different algorithms when creating output chunks.
   rechunk:
     method: "tasks"  # Rechunking method to use
     threshold: 4
-  shuffle:
-    chunksize-tolerance: 1.25  # Tolerance for the shuffle algorithm when creating output chunks.
   svg:
     size: 120  # pixels
   slicing:


Test patch:
diff --git a/dask/array/tests/test_shuffle.py b/dask/array/tests/test_shuffle.py
index 86cadcfc2..f0f717de1 100644
--- a/dask/array/tests/test_shuffle.py
+++ b/dask/array/tests/test_shuffle.py
@@ -40,7 +40,7 @@ def test_shuffle(arr, darr, indexer, chunks, other_chunks):
 @pytest.mark.parametrize("tol, chunks", ((1, (3, 2, 3)), (1.4, (5, 3))))
 def test_shuffle_config_tolerance(arr, darr, tol, chunks):
     indexer = [[1, 5, 6], [0, 3], [4, 2, 7]]
-    with dask.config.set({"array.shuffle.chunksize-tolerance": tol}):
+    with dask.config.set({"array.chunk-size-tolerance": tol}):
         result = darr.shuffle(indexer, axis=1)
     expected = arr[:, [1, 5, 6, 0, 3, 4, 2, 7]]
     assert_eq(result, expected)