Repository: dask/dask

Issue:
Better error message for unsupported Array reshape operations
`Array.reshape` "only allows for reshapings that collapse or merge dimensions" (xref https://github.com/dask/dask/issues/2561). However, when you try to do one of these unsupported reshapings, the error message `Shapes not compatible` does not make it at all clear that what you're asking for just isn't supported by dask. Instead, it sounds as though your inputs are invalid.

A more descriptive error message, possibly with a link to documentation like https://docs.dask.org/en/latest/array-chunks.html#reshaping, would help users understand what's going on. Also, this should maybe be a NotImplementedError.


```python
import dask.array as da

# NumPy works fine
In [25]: da.random.random((6, 2), chunks=(2, 2)).compute().reshape(3, 4)
Out[25]: 
array([[0.79357568, 0.05108054, 0.08983493, 0.197254  ],
       [0.40013843, 0.43548821, 0.99473879, 0.02848385],
       [0.82485068, 0.21818596, 0.65396113, 0.20300868]])

# Dask does not
In [26]: da.random.random((6, 2), chunks=(2, 2)).reshape(3, 4)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-26-2733f73e03ee> in <module>
----> 1 da.random.random((6, 2), chunks=(2, 2)).reshape(3, 4)

~/dev/dask/dask/array/core.py in reshape(self, merge_chunks, limit, *shape)
   2080         if len(shape) == 1 and not isinstance(shape[0], Number):
   2081             shape = shape[0]
-> 2082         return reshape(self, shape, merge_chunks=merge_chunks, limit=limit)
   2083 
   2084     def topk(self, k, axis=-1, split_every=None):

~/dev/dask/dask/array/reshape.py in reshape(x, shape, merge_chunks, limit)
    233         x = x.rechunk({i: 1 for i in range(din - dout)})
    234 
--> 235     inchunks, outchunks = reshape_rechunk(x.shape, shape, x.chunks)
    236     # Check output chunks are not too large
    237     max_chunksize_in_bytes = reduce(mul, [max(i) for i in outchunks]) * x.dtype.itemsize

~/dev/dask/dask/array/reshape.py in reshape_rechunk(inshape, outshape, inchunks)
     45                 ileft -= 1
     46             if reduce(mul, inshape[ileft : ii + 1]) != dout:
---> 47                 raise ValueError("Shapes not compatible")
     48 
     49             # Special case to avoid intermediate rechunking:

ValueError: Shapes not compatible
```


Discussion:
can i work on this issue?

@gjoseph92  so the message  should be simply "Not Implemented" right?
The exception should be a `NotImplementedError`. Coming up with a good, descriptive error message that helps users is part of the task, though I could write one if you'd like.

Note that this also isn't as simple as changing the text in that one line of code. You need to read and understand the reshape code first to understand if this error is _also_ currently catching cases where you've actually passed in an invalid shape, as opposed to a shape which _could_ be valid, but dask just doesn't support right now. Then you'd also need to figure out how to differentiate between those cases.
ok i will work on it.
 
@gjoseph92 i have noticed that if we pass invalid shape it gives error message shape of new array must be unchanged which is correct. so i need to change message as NotImplemented?
can you provide an error message if possible?
@Dhruv454000 Thanks for your interest in this! Would you be comfortable opening a PR where we can continue this discussion? It'll be easier for folks to share in-line feedback that way :)
Is this issue solved or can I fix it as a first issue? :)
Hi @TristanBilot, feel free to pick it up, it has not been fixed yet!

Resolving patch:
diff --git a/dask/array/reshape.py b/dask/array/reshape.py
--- a/dask/array/reshape.py
+++ b/dask/array/reshape.py
@@ -14,6 +14,20 @@
 from dask.highlevelgraph import HighLevelGraph
 from dask.utils import M, parse_bytes
 
+_not_implemented_message = """
+Dask's reshape only supports operations that merge or split existing dimensions
+evenly. For example:
+
+>>> x = da.ones((6, 5, 4), chunks=(3, 2, 2))
+>>> x.reshape((3, 2, 5, 4))  # supported, splits 6 into 3 & 2
+>>> x.reshape((30, 4))       # supported, merges 6 & 5 into 30
+>>> x.reshape((4, 5, 6))     # unsupported, existing dimensions split unevenly
+
+To work around this you may call reshape in multiple passes, or (if your data
+is small enough) call ``compute`` first and handle reshaping in ``numpy``
+directly.
+"""
+
 
 def reshape_rechunk(inshape, outshape, inchunks):
     assert all(isinstance(c, tuple) for c in inchunks)
@@ -44,8 +58,7 @@ def reshape_rechunk(inshape, outshape, inchunks):
             ):  # 4 < 64, 4*4 < 64, 4*4*4 == 64
                 ileft -= 1
             if reduce(mul, inshape[ileft : ii + 1]) != dout:
-                raise ValueError("Shapes not compatible")
-
+                raise NotImplementedError(_not_implemented_message)
             # Special case to avoid intermediate rechunking:
             # When all the lower axis are completely chunked (chunksize=1) then
             # we're simply moving around blocks.
@@ -74,8 +87,7 @@ def reshape_rechunk(inshape, outshape, inchunks):
             while oleft >= 0 and reduce(mul, outshape[oleft : oi + 1]) < din:
                 oleft -= 1
             if reduce(mul, outshape[oleft : oi + 1]) != din:
-                raise ValueError("Shapes not compatible")
-
+                raise NotImplementedError(_not_implemented_message)
             # TODO: don't coalesce shapes unnecessarily
             cs = reduce(mul, outshape[oleft + 1 : oi + 1])
 


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -1,6 +1,7 @@
 import contextlib
 import copy
 import pathlib
+import re
 import xml.etree.ElementTree
 from unittest import mock
 
@@ -47,6 +48,7 @@
     stack,
     store,
 )
+from dask.array.reshape import _not_implemented_message
 from dask.array.tests.test_dispatch import EncapsulateNDArray
 from dask.array.utils import assert_eq, same_keys
 from dask.base import compute_as_if_collection, tokenize
@@ -1212,14 +1214,13 @@ def test_reshape_splat():
     assert_eq(x.reshape((25,)), x.reshape(25))
 
 
-def test_reshape_fails_for_dask_only():
-    cases = [((3, 4), (4, 3), 2)]
-    for original_shape, new_shape, chunks in cases:
-        x = np.random.randint(10, size=original_shape)
-        a = from_array(x, chunks=chunks)
-        assert x.reshape(new_shape).shape == new_shape
-        with pytest.raises(ValueError):
-            da.reshape(a, new_shape)
+def test_reshape_not_implemented_error():
+    a = da.ones((4, 5, 6), chunks=(2, 2, 3))
+    for new_shape in [(2, 10, 6), (5, 4, 6), (6, 5, 4)]:
+        with pytest.raises(
+            NotImplementedError, match=re.escape(_not_implemented_message)
+        ):
+            a.reshape(new_shape)
 
 
 def test_reshape_unknown_dimensions():