Repository: dask/dask

Issue:
Series comparison to NumPy scalar fails
Comparing a dask Series with a NumPy scalar or 0D array fails in certain cases. Here is a minimal reproducible example:

```python
import dask.dataframe as dd
import pandas as pd

np.float64(5.2) >= dd.from_array(np.arange(10))
```

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-18-d905075c0565> in <module>()
      1 
----> 2 np.float64(10) >= dd.from_array(np.arange(10))

~/miniconda/envs/anacondaviz/lib/python3.6/site-packages/dask/dataframe/core.py in __array_wrap__(self, array, context)
   1704     def __array_wrap__(self, array, context=None):
   1705         if isinstance(context, tuple) and len(context) > 0:
-> 1706             index = context[1][0].index
   1707 
   1708         return pd.Series(array, index=index, name=self.name)

AttributeError: 'numpy.ndarray' object has no attribute 'index'
```

I can reproduce this using all releases in the dask 0.17.x series.


Discussion:
cc @martindurant this might be a good issue if you have time

On Tue, Apr 10, 2018 at 11:18 AM, Philipp Rudiger <notifications@github.com>
wrote:

> Comparing a dask Series with a NumPy scalar or 0D array fails in certain
> cases. Here is a minimal reproducible example:
>
> import dask.dataframe as ddimport pandas as pd
>
> np.float64(5.2) >= dd.from_array(np.arange(10))
>
> ---------------------------------------------------------------------------
> AttributeError                            Traceback (most recent call last)
> <ipython-input-18-d905075c0565> in <module>()
>       1
> ----> 2 np.float64(10) >= dd.from_array(np.arange(10))
>
> ~/miniconda/envs/anacondaviz/lib/python3.6/site-packages/dask/dataframe/core.py in __array_wrap__(self, array, context)
>    1704     def __array_wrap__(self, array, context=None):
>    1705         if isinstance(context, tuple) and len(context) > 0:
> -> 1706             index = context[1][0].index
>    1707
>    1708         return pd.Series(array, index=index, name=self.name)
>
> AttributeError: 'numpy.ndarray' object has no attribute 'index'
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3392>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszHQV1Rp5uH9jCguuPd_Yt7-FYMb-ks5tnNtpgaJpZM4TOlMB>
> .
>

I also get an error for the same code, but not the same error. Running from master.

```
In [20]: np.float64(5.2) >= dd.from_array(np.arange(10))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-20-f79e026dfad4> in <module>()
----> 1 np.float64(5.2) >= dd.from_array(np.arange(10))

TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'greater_equal'>, '__call__', array(5.2), Dask Series Structure:
npartitions=1
0    int64
9      ...
dtype: int64
Dask Name: from_array, 1 tasks): 'ndarray', 'Series'

In [22]: np.__version__
Out[22]: '1.13.3'

In [23]: dask.__version__
Out[23]: '0.17.2+49.g7cd5143'
```

I'm getting the same error as @martindurant. 

It looks like `np.float64(5.2)` is getting converted to a 0-dimensional array (i.e. `np.array(5.2)`) when inputted into `__array_ufunc__`. Because of this, the `isinstance` check for `Number` fails

https://github.com/dask/dask/blob/b64e694032ee43273fafd47359df5237af1dacb0/dask/dataframe/core.py#L347-L349

and an error is thrown. 

 I can look into adding an extra check in `__array_ufunc__` for 0-dimensional `np.ndarray`s 
If anyone has time to fix this that would be welcome.

On Mon, Apr 23, 2018 at 6:13 PM, James Bourbeau <notifications@github.com>
wrote:

> I'm getting the same error as @martindurant
> <https://github.com/martindurant>.
>
> It looks like np.float64(5.2) is getting converted to a 0-dimensional
> array (i.e. np.array(5.2)) when inputted into __array_ufunc__. Because of
> this, the isinstance check for Number fails
>
> https://github.com/dask/dask/blob/b64e694032ee43273fafd47359df52
> 37af1dacb0/dask/dataframe/core.py#L347-L349
>
> and an error is thrown.
>
> I can look into adding an extra check in __array_ufunc__ for
> 0-dimensional np.ndarrays
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3392#issuecomment-383739840>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszOF1wnTNV_W_DuQj9Iu0FE9QlgOJks5trlIDgaJpZM4TOlMB>
> .
>

I can take this one up 

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -344,8 +344,12 @@ def __array_wrap__(self, array, context=None):
     def __array_ufunc__(self, numpy_ufunc, method, *inputs, **kwargs):
         out = kwargs.get('out', ())
         for x in inputs + out:
-            if not isinstance(x, (Number, Scalar, _Frame, Array,
-                                  pd.DataFrame, pd.Series, pd.Index)):
+            # ufuncs work with 0-dimensional NumPy ndarrays
+            # so we don't want to raise NotImplemented
+            if isinstance(x, np.ndarray) and x.shape == ():
+                continue
+            elif not isinstance(x, (Number, Scalar, _Frame, Array,
+                                    pd.DataFrame, pd.Series, pd.Index)):
                 return NotImplemented
 
         if method == '__call__':
@@ -1751,7 +1755,10 @@ class Series(_Frame):
 
     def __array_wrap__(self, array, context=None):
         if isinstance(context, tuple) and len(context) > 0:
-            index = context[1][0].index
+            if isinstance(context[1][0], np.ndarray) and context[1][0].shape == ():
+                index = None
+            else:
+                index = context[1][0].index
 
         return pd.Series(array, index=index, name=self.name)
 
@@ -2311,7 +2318,10 @@ class DataFrame(_Frame):
 
     def __array_wrap__(self, array, context=None):
         if isinstance(context, tuple) and len(context) > 0:
-            index = context[1][0].index
+            if isinstance(context[1][0], np.ndarray) and context[1][0].shape == ():
+                index = None
+            else:
+                index = context[1][0].index
 
         return pd.DataFrame(array, index=index, columns=self.columns)
 


Test patch:
diff --git a/dask/dataframe/tests/test_ufunc.py b/dask/dataframe/tests/test_ufunc.py
--- a/dask/dataframe/tests/test_ufunc.py
+++ b/dask/dataframe/tests/test_ufunc.py
@@ -374,3 +374,18 @@ def test_ufunc_with_reduction(redfunc, ufunc, pandas):
     with pytest.warns(None):
         assert isinstance(np_redfunc(dask), (dd.DataFrame, dd.Series, dd.core.Scalar))
         assert_eq(np_redfunc(np_ufunc(dask)), np_redfunc(np_ufunc(pandas)))
+
+
+@pytest.mark.parametrize('pandas',
+                         [pd.Series(np.random.randint(1, 100, size=100)),
+                          pd.DataFrame({'A': np.random.randint(1, 100, size=20),
+                                        'B': np.random.randint(1, 100, size=20),
+                                        'C': np.abs(np.random.randn(20))})])
+@pytest.mark.parametrize('scalar', [15, 16.4, np.int64(15), np.float64(16.4)])
+def test_ufunc_numpy_scalar_comparison(pandas, scalar):
+    # Regression test for issue #3392
+
+    dask_compare = scalar >= dd.from_pandas(pandas, npartitions=3)
+    pandas_compare = scalar >= pandas
+
+    assert_eq(dask_compare, pandas_compare)