Repository: dask/dask

Issue:
dask mistakenly thinks or tries to invert a boolean as if it is a float.
As part of an application I am trying to compare the values of a column to a constant and keep track of the ones that I want to act on. I am getting the following error:

`E           TypeError: ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''`

I have created a minimal reproducible example. First let's set up a new environment:

```
# We are starting from a Fedora 31 system that uses python 3.7.5 by default
# python 3.7 is recommended when trying to reproduce this bug
$ python3 -m venv dask
# activate virtual environment
$ source dask/bin/activate
# install required packages
(dask) $ pip install --upgrade pip
(dask) $ pip install "dask[dataframe]"
(dask) $ pip install pytest
```

We save as `dask_inv_error.py` the following file:

```
import pandas as pd
import dask.dataframe as dd
import numpy as np


def test_mre():
    """
    Minimum reproducible example for dask error
    E   TypeError: ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced
    to any supported types according to the casting rule ''safe''
    """

    # setting up
    datadict = {"cindex": range(5), "COL": [1, 2, 10, -1, -9]}
    pdf1 = pd.DataFrame(datadict)
    ddf = dd.from_pandas(pdf1, npartitions=4)

    ddf["COL1"] = np.False_
    ddf["COL2"] = np.False_

    val = float(0)
    sdf1 = ddf["COL"] > val

    # Probable cause is that sdf1 gets assigned 'COL' name
    print(f"sdf1 name is: {sdf1.compute().name}")
    #  Fix1: If we were to change the column name we don't get the error
    # sdf1 = sdf1.rename("fix1")

    sdf2 = ~ddf["COL2"]
    # Computation that is problematic below - comment out to apply fixes 2, 3a, 3b
    ddf["COL1"] = ddf["COL1"].mask(cond=sdf2, other=sdf1)
    # The problem comes from the computations resulting in  sdf1
    # Fix2: Running the code below instead of the line above sidesteps the problem
    # ddf["COL1"] = ddf["COL1"].partitions[:].mask(cond=sdf2, other=sdf1)
    # Fix3a: Replacing problematic computation time with the one below sidesteps the problem
    # ddf["COL1"] = ddf["COL1"].loc[:].mask(cond=sdf2, other=sdf1)
    # Fix3b: Replacing problematic computation time with the one below sidesteps the problem
    # ddf["COL1"] = ddf.loc[:, "COL1"].mask(cond=sdf2, other=sdf1)

    print(f"ddf is:\n{ddf.compute()}")
    print(f"ddf types are:\n{ddf.compute().dtypes}")


if __name__ == "__main__":
    test_mre()
```

We can run the following code with:

```
(dask) $ python dask_inv_error.py
```

or:

```
(dask) $ pytest dask_inv_error.py
# ...
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = 0    0.0
1    0.0
Name: COL2, dtype: float64

    def __invert__(self):
        try:
>           arr = operator.inv(com.values_from_object(self))
E           TypeError: ufunc 'invert' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

dask/lib64/python3.7/site-packages/pandas/core/generic.py:1541: TypeError
---------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
sdf1 name is: COL
================================================================================================ 1 failed in 0.58s ================================================================================================
```

I have only left the last part of the error. **It reports `ddf["COL2"] ` as `float64` even though it should be `bool` and dask should be able to invert it and perform the computation.** It actually succeeds in doing so once any of the fixes in the code are applied in order to sidestep the problem.

As you can see in the code there are 3 ways to sidestep the bug that causes the error:

- Fix1 is done by renaming the `sdf1` series. To do that just uncomment the relevant line of code from the mre. This suggests that the issue could be caused by a name collision between `sdf1`'s name and the `ddf['COL']` column names that results in inappropriate column type inferences. `sdf1` gets named by dask and the collision somehow!? causes it to change `ddf2['COL2']` type to float.
- Fixes 2, 3a and 3b can be used by commenting out the problematic line of code and uncommenting the line of code relating to each Fix. These fixes involve .partitions or .loc and imply that something unexpected is happening on the actual computation when they are not used.

Note: In case it becomes important over time here are the versions of packages used in the python 3.7.5 environment that reproduces this bug:

```
$ pip list
Package            Version
------------------ -------
attrs              19.3.0 
dask               2.9.0  
fsspec             0.6.1  
importlib-metadata 1.2.0  
locket             0.2.0  
more-itertools     8.0.2  
numpy              1.17.4 
packaging          19.2   
pandas             0.25.3 
partd              1.1.0  
pip                19.3.1 
pluggy             0.13.1 
py                 1.8.0  
pyparsing          2.4.5  
pytest             5.3.1  
python-dateutil    2.8.1  
pytz               2019.3 
setuptools         41.2.0 
six                1.13.0 
toolz              0.10.0 
wcwidth            0.1.7  
zipp               0.6.0  
```


Discussion:
The issue appears to be related to the assignment of `COL1` and `COL2` on `ddf`. If I handle that on the pandas dataframe beforehand then everything works fine:

```python
import pandas as pd
import numpy as np

datadict = {
    "cindex": range(5),
    "COL": [1, 2, 10, -1, -9],
    "COL1": False,
    "COL2": False,
}
pdf1 = pd.DataFrame(datadict)
ddf = dd.from_pandas(pdf1, npartitions=4)

cond = ~ddf["COL2"]
other = ddf["COL"] > 0.0
ddf["COL1"] = ddf["COL1"].mask(cond=cond, other=other)

print(f"ddf is:\n{ddf.compute()}")
print(f"ddf types are:\n{ddf.compute().dtypes}")
```

Debugging now.
I've tracked this down to a bug in `rewrite_blockwise` (one of our optimization routines). The bug is causing False to be swapped out for `0.0`, since in Python the following holds true:

```
>>> False == 0.0
True
```

If you turn off optimizations (pass `optimize_graph=False` to `.compute()`), or use a value that isn't 0 (e.g. `ddf["COL"] > 0.1) then everything works fine.

Working on a fix now.

Resolving patch:
diff --git a/dask/blockwise.py b/dask/blockwise.py
--- a/dask/blockwise.py
+++ b/dask/blockwise.py
@@ -643,15 +643,14 @@ def rewrite_blockwise(inputs):
 
             # Bump new inputs up in list
             sub = {}
+            # Map from (id(key), inds or None) -> index in indices. Used to deduplicate indices.
+            index_map = {(id(k), inds): n for n, (k, inds) in enumerate(indices)}
             for i, index in enumerate(new_indices):
-                try:
-                    contains = index in indices
-                except (ValueError, TypeError):
-                    contains = False
-
-                if contains:  # use old inputs if available
-                    sub[blockwise_token(i)] = blockwise_token(indices.index(index))
+                id_key = (id(index[0]), index[1])
+                if id_key in index_map:  # use old inputs if available
+                    sub[blockwise_token(i)] = blockwise_token(index_map[id_key])
                 else:
+                    index_map[id_key] = len(indices)
                     sub[blockwise_token(i)] = blockwise_token(len(indices))
                     indices.append(index)
             new_dsk = subs(inputs[dep].dsk, sub)
@@ -659,8 +658,6 @@ def rewrite_blockwise(inputs):
             # indices.extend(new_indices)
             dsk.update(new_dsk)
 
-    indices = [(a, tuple(b) if isinstance(b, list) else b) for a, b in indices]
-
     # De-duplicate indices like [(a, ij), (b, i), (a, ij)] -> [(a, ij), (b, i)]
     # Make sure that we map everything else appropriately as we remove inputs
     new_indices = []


Test patch:
diff --git a/dask/array/tests/test_atop.py b/dask/array/tests/test_atop.py
--- a/dask/array/tests/test_atop.py
+++ b/dask/array/tests/test_atop.py
@@ -141,6 +141,27 @@
                 [(456, None), (a, "j"), (123, None)],
             ),
         ],
+        # Literals that compare equal (e.g. 0 and False) aren't deduplicated
+        [
+            [
+                (b, "i", {b: (add, _0, _1)}, [(a, "i"), (0, None)]),
+                (c, "j", {c: (add, _0, _1)}, [(b, "j"), (False, None)]),
+            ],
+            (
+                c,
+                "j",
+                {b: (add, _1, _2), c: (add, b, _0)},
+                [(False, None), (a, "j"), (0, None)],
+            ),
+        ],
+        # Literals are deduplicated
+        [
+            [
+                (b, "i", {b: (add, _0, _1)}, [(a, "i"), (123, None)]),
+                (c, "j", {c: (add, _0, _1)}, [(b, "j"), (123, None)]),
+            ],
+            (c, "j", {b: (add, _1, _0), c: (add, b, _0)}, [(123, None), (a, "j")]),
+        ],
     ],
 )
 def test_rewrite(inputs, expected):