Repository: numba/numba
Pull request: better error messages for np.transpose and tuples
<!--

Thanks for wanting to contribute to Numba :)

First, if you need some help or want to chat to the core developers, please
visit https://gitter.im/numba/numba for real time chat or post to the Numba
mailing list https://groups.google.com/a/continuum.io/forum/#!forum/numba-users.

Here's some guidelines to help the review process go smoothly.

0. Please write a description in this text box of the changes that are being
   made.

1. Please ensure that you have written units tests for the changes made/features
   added.

2. If you are closing an issue please use one of the automatic closing words as
   noted here: https://help.github.com/articles/closing-issues-using-keywords/

3. If your pull request is not ready for review but you want to make use of the
   continuous integration testing facilities please label it with `[WIP]` and
   then remove the label when you'd like it to be reviewed.

4. Once review has taken place please do not add features or make changes out of
   the scope of those requested by the reviewer (doing this just add delays as
   already reviewed code ends up having to be re-reviewed/it is hard to tell
   what is new etc!). Further, please do not rebase your branch on master/force
   push/rewrite history, doing any of these causes the context of any comments
   made by reviewers to be lost. If conflicts occur against master they should
   be resolved by merging master into the branch used for making the pull
   request.

Many thanks in advance for your cooperation!

-->

Fixes #4053

NumPy 'transpose' does accept tuples but Numba doesn't and there is a
good reason for this. The problem is, that we don't know the shape of
the arrays contained within the tuple during the type inference phase.
If they have inhomogeneous shapes, we may end up with a ragged/jagged
array. We could check all tuples and their shapes at runtime but that
will be cumbersome. Instead, we issue a better warning instead for now.

For those wanting to use `np.argwhere` using the following snippet:

```
np.transpose(np.nonzero(a))
```

The appropriate workaround is:

```
np.transpose(np.vstack(np.nonzero(a)))
```

Problem statement:
np.transpose crashes when passed a tuple
Minimal reproducing code using `numba==0.43.1`:
```
from numba import jit
import numpy as np


def without_numba():
    t = (np.array([0, 1]),)
    print(np.transpose(t))

@jit(nopython=True)
def with_numba():
    t = (np.array([0, 1]),)
    print(np.transpose(t))


without_numba()
with_numba()
```

The error reported mentions:
```
Invalid use of Function(<function transpose at 0x08479ED0>) with argument(s) of type(s): (tuple(array(int32, 1d, C) x 1))
```

I tried to find a fix, e.g. attempted to add a call to `np.asarray` in `numpy_transpose` in `targets/arrayobj.py:1492`, but no luck. It actually revealed a second bug:

```
def without_numba():
    t = (np.array([0, 1]),)
    print(np.asarray(t))

@jit(nopython=True)
def with_numba():
    t = (np.array([0, 1]),)
    print(np.asarray(t))
```

I am a bit overwhelmed by Numba's code complexity, so I don't think I'll be able to find a fix on my own.

Hints from issue discussion:
@Lucas-C thanks for using Numba and reporting this. I can reproduce this locally and it looks like a bug indeed. The Numpy implementation seems to be able to deal with tuples that contain arrays, whereas the Numba one doesn't.
@esc agree, the problem is fundamentally that `np.transpose` is wired through from `ndarray.transpose` in Numba, and there's a `tuple-of-array` type being passed which is not handled.
```python
from numba import jit
import numpy as np

@jit(nopython=True)
def with_numba():
    t = (np.array([0, 1]),) # <--- tuple(array,)
    print(np.transpose(t))  # <- transpose impl only takes arrays! There's no `tuple.transpose()`


with_numba.py_func()
with_numba()
```
adding a `np.asarray` does indeed expose a second issue in that it also doesn't handle `tuple-of-array`, e.g.:
```python
@jit(nopython=True)
def foo():
    return np.asarray((np.array([0, 1]),))

foo()
```
I think fixing the latter would permit fixing the former.

@Lucas-C so, I thought about this and spoke to my colleague, and I have come to the conclusion, that this will probably not be supported anytime soon. The main problem with this is, that in order to get a proper array from the initial tuple, we need to ensure that all the arrays contained within have the same length/size/shape. Unfortunately, this information isn't available at the typing stage, only information about the type and about the dimensionality is available then. We could do the conversion at runtime, but that will be quite hacky. Also, if you look at what NumPy does in case the arrays are not of the same length:

```
In [1]: import numpy as np

In [2]: a = (np.zeros(2), np.zeros(1))

In [3]: a
Out[3]: (array([0., 0.]), array([0.]))

In [4]: np.transpose(a)
Out[4]: array([array([0., 0.]), array([0.])], dtype=object)

In [5]: a = (np.zeros(2), np.zeros(2))

In [6]: np.transpose(a)
Out[6]:
array([[0., 0.],
       [0., 0.]])
```

As you can see, in the first case, NumPy will convert the tuple to an `object` array. I should mention that Numba has zero support for the `object` dtype.

What can be done, however, is improve the error message, and ensure that `np.transpose` doesn't accept tuples and issues an appropriate error message. The respective code is here:

https://github.com/numba/numba/blob/master/numba/targets/arrayobj.py#L1491

And there is some preliminary documentation here:

https://github.com/numba/numba/pull/3951

Lastly, to solve you original issue, I would suggest to make a 2-D array in the first place and then handle that. If that is not a possibility, maybe you could tell us a little more about your use-case and we will see if we can find a workaround for you?
Thank you very much for your work and this and detailed explanation !

My initial use case was to use [`np.argwhere`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.argwhere.html). Because it is not supported by numba currently and [under the hood](https://github.com/numpy/numpy/blob/v1.16.1/numpy/core/numeric.py#L842-L881) it calls `transpose(nonzero(a))`, I attempted to call those. But `np.nonzero` returns a tuple, hence this issue...
@Lucas-C can you wrap the output in a call to `np.asarray` from outside of JIT compiled code?
@Lucas-C I see where you are coming from, I just tried the following:

```
In [7]: @njit
   ...: def foo(a):
   ...:     return np.transpose(np.nonzero(a))
```

And that leads exactly to the issue you reported.
For reference the tuple is created here:

https://github.com/numba/numba/blob/master/numba/targets/arraymath.py#L2646
@Lucas-C you can use the following, potentially:

```
In [11]: @njit
    ...: def foo(a):
    ...:     return np.transpose(np.vstack(np.nonzero(a)))
```
Thank you @esc !
I missed the notification that you replied here, sorry.
Thanks for the suggestion. I won't be able to test it right now but it looks like a nice workaround.
As far as I'm concern, it fits my need
@Lucas-C no problem, glad I could help. @stuartarchibald shall we close this one?

Commit message: better error messages for np.transpose and tuples

Fixes #4053

NumPy 'transpose' does accept tuples but Numba doesn't and there is a
good reason for this. The problem is, that we don't know the shape of
the arrays contained within the tuple during the type inference phase.
If they have inhomogeneous shapes, we may end up with a ragged/jagged
array. We could check all tuples and their shapes at runtime but that
will be cumbersome. Instead, we issue a better warning instead for now.

For those wanting to use `np.argwhere` using the following snippet:

```
np.transpose(np.nonzero(a))
```

The appropriate workaround is:

```
np.transpose(np.vstack(np.nonzero(a)))
```

Patch to review:
diff --git a/numba/targets/arrayobj.py b/numba/targets/arrayobj.py
index 5878d4bf31c..503e20691a5 100644
--- a/numba/targets/arrayobj.py
+++ b/numba/targets/arrayobj.py
@@ -1491,6 +1491,8 @@ def array_transpose_vararg(context, builder, sig, args):
 
 @overload(np.transpose)
 def numpy_transpose(a, axes=None):
+    if isinstance(a, types.BaseTuple):
+        raise errors.TypingError("np.transpose does not accept tuples")
 
     if axes is None:
         def np_transpose_impl(a, axes=None):
diff --git a/numba/tests/test_array_manipulation.py b/numba/tests/test_array_manipulation.py
index ddc27d6fbe8..551b87c0751 100644
--- a/numba/tests/test_array_manipulation.py
+++ b/numba/tests/test_array_manipulation.py
@@ -333,6 +333,10 @@ def check_err_invalid_args(pyfunc, arr, axes):
         check_err_axis_oob(arrs[3], (3, 1, 2, 5))
         check_err_axis_oob(arrs[3], (3, 1, 2, -5))
 
+        with self.assertRaises(TypingError) as e:
+            jit(nopython=True)(numpy_transpose_array)((np.array([0, 1]),))
+        self.assertIn("np.transpose does not accept tuples",
+                        str(e.exception))
 
     @tag('important')
     def test_expand_dims(self):


Code review comments:
diff_hunk: @@ -1491,6 +1491,8 @@ def array_transpose_vararg(context, builder, sig, args):
 
 @overload(np.transpose)
 def numpy_transpose(a, axes=None):
+    if isinstance(a, types.BaseTuple):
+        raise errors.TypingError("np.transpose does not accept tuples")
path: numba/targets/arrayobj.py
text: @user1:
Should probably raise an `error.UnsupportedError` ? It's not that it cannot be typed/it's bad, it's just Numba won't handle it.

@author:
check