Repository: dask/dask

Issue:
Provide byte size in chunks keyword?
How do people feel about the following:

```python
x = da.ones(1000000000, chunks="256 MiB")
```

This would use the `"auto"` chunking mechanism, and use the provided value as the desired chunk size.


Discussion:
Planning to interpret silence here as assent :)
I was about to suggest a similar parameter for `dataframe.repartition` to basically implement the approach you suggested in [this SO question](https://stackoverflow.com/questions/44657631/strategy-for-partitioning-dask-dataframes-efficiently). Should I open a new issue for it?
Yes, a different issue would be good here.

On Mon, Jan 21, 2019 at 10:37 AM George Sakkis <notifications@github.com>
wrote:

> I was about to suggest a similar parameter for dataframe.repartition to
> basically implement the approach you suggested in this SO question
> <https://stackoverflow.com/questions/44657631/strategy-for-partitioning-dask-dataframes-efficiently>.
> Should I open a new issue for it?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/4397#issuecomment-456165623>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszHS0K_bN_wcDcuRK9Lkejxyw6-jbks5vFgjpgaJpZM4aEL29>
> .
>

In the case of a 1-D array the meaning of this is somewhat clear. However it's less clear what this means for arbitrary dimensionality. Could you please elaborate?
We would use the same logic as is currently implemented for the "auto"
chunking value

On Tue, Jan 22, 2019 at 4:14 AM jakirkham <notifications@github.com> wrote:

> In the case of a 1-D array the meaning of this is somewhat clear. However
> it's less clear what this means for arbitrary dimensionality. Could you
> please elaborate?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/4397#issuecomment-456378554>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszJZtoRwK3EuumtSxk1HBPVRI1aBzks5vFwC3gaJpZM4aEL29>
> .
>

Some relevant pieces of code for this

https://github.com/dask/dask/blob/9241a2ae16c0eb54073b16bc3ae01bf40f43c90c/dask/utils.py#L950-L997

https://github.com/dask/dask/blob/9241a2ae16c0eb54073b16bc3ae01bf40f43c90c/dask/array/core.py#L1776-L1897

note in particular the `chunks` keyword (which will be something like "10 MiB") and the `limit=` keyword, which we use within the function to determine the ideal chunksize 
I spoke to @beberg offline and he seemed interested in doing this
It might be that the user provides conflicting or overlapping values for `limit=` or byte chunk values like 

```python
x = da.ones((100000, 100000), chunks=('1MiB', '10 MiB'))
```

In these cases I suggest that we raise a ValueError encouraging them to specify at most one byte size value.

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -1868,6 +1868,20 @@ def normalize_chunks(chunks, shape=None, limit=None, dtype=None,
     if -1 in chunks:
         chunks = tuple(s if c == -1 else c for c, s in zip(chunks, shape))
 
+    # If specifying chunk size in bytes, use that value to set the limit.
+    # Verify there is only one consistent value of limit or chunk-bytes used.
+    for c in chunks:
+        if isinstance(c, str) and c != 'auto':
+            parsed = parse_bytes(c)
+            if limit is None:
+                limit = parsed
+            elif parsed != limit:
+                raise ValueError(
+                    "Only one consistent value of limit or chunk is allowed."
+                    "Used %s != %s" % (parsed, limit))
+    # Substitute byte limits with 'auto' now that limit is set.
+    chunks = tuple('auto' if isinstance(c, str) and c != 'auto' else c for c in chunks)
+
     if any(c == 'auto' for c in chunks):
         chunks = auto_chunks(chunks, shape, limit, dtype, previous_chunks)
 


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -3647,3 +3647,25 @@ def func(a, b):
 
     assert_eq(da.map_blocks(func, x, y, chunks=(2,), dtype=x.dtype),
               np.array([99, 9, 199, 19, 299, 29, 399, 39]))
+
+
+def test_nbytes_auto():
+    chunks = normalize_chunks("800B", shape=(500,), dtype='float64')
+    assert chunks == ((100, 100, 100, 100, 100),)
+    chunks = normalize_chunks("200B", shape=(10, 10), dtype='float64')
+    assert chunks == ((5, 5), (5, 5))
+    chunks = normalize_chunks((5,"200B"), shape=(10, 10), dtype='float64')
+    assert chunks == ((5, 5), (5, 5))
+    chunks = normalize_chunks("33B", shape=(10, 10), dtype='float64')
+    assert chunks == ((2, 2, 2, 2, 2), (2, 2, 2, 2, 2))
+    chunks = normalize_chunks("1800B", shape=(10, 20, 30), dtype='float64')
+    assert chunks == ((5, 5), (5, 5, 5, 5), (6, 6, 6, 6, 6))
+
+    with pytest.raises(ValueError):
+        normalize_chunks("10B", shape=(10,), limit=20, dtype='float64')
+    with pytest.raises(ValueError):
+        normalize_chunks("100B", shape=(10, 10), limit=20, dtype='float64')
+    with pytest.raises(ValueError):
+        normalize_chunks(("100B", "10B"), shape=(10, 10), dtype='float64')
+    with pytest.raises(ValueError):
+        normalize_chunks(("10B", "10B"), shape=(10, 10), limit=20, dtype='float64')