Repository: dask/dask

Issue:
[upstream/build] Groupby mean result index doesn't preserve bool dtype
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:

@ncclementi and I were tracking down this error on her fork with upstream dependencies:
https://github.com/ncclementi/dask/runs/5148303425?check_suite_focus=true#step:5:11490

Following a recent pandas bug enhancement (https://github.com/pandas-dev/pandas/pull/45061), groupby results with bool-typed indexes are better respected. In dask, the index type generally remains `bool`, but `groupby.mean()` returns a dataframe with an `object`-dtyped index.

If you look at the example below, most operations return a bool index. Mean returns an object index. Interestingly, doing mathematical operations with two dataframes with bool-typed indexes returns a dataframe with an object-typed index

cc: @jrbourbeau 
We were not able to track down where the fix for this would occur and need some guidance.

**What you expected to happen**:

`Groupby.mean` should preserve the index dtypes, which in turn will allow the tests to pass 

**Minimal Complete Verifiable Example**:

```python
import pandas as pd
import dask.dataframe as dd

pdf = pd.DataFrame(
            {"a": [1, 2, 6, 4, 4, 6, 4, 3, 7], "b": [4, 2, 7, 3, 3, 1, 1, 1, 2]},
            index=[0, 1, 3, 5, 6, 8, 9, 9, 9],
        )
ddf = dd.from_pandas(pdf, npartitions=3)

gb = ddf.groupby(ddf.b > 4).a

assert gb.count().index.dtype == bool  # good
assert gb.sum().index.dtype == bool  # good
assert gb.mean().index.dtype == object  # bad (implemented as sum() / count(), see link below)
## and also:
assert (gb.sum() / gb.count()).index.dtype == object  # bad

```

https://github.com/dask/dask/blob/d670db6f58e47e8b0a5019f75330121dda11af84/dask/dataframe/groupby.py#L1424-L1429

**Anything else we need to know?**:

**Environment**:

- Dask version: main (d670db6f58e47e8b0a5019f75330121dda11af84)
- Pandas: upstream (1.5.0.dev0+350.g0631637205)
- Python version: 3.9.10
- Operating System: Mac OS 12.2.1 (21D62)
- Install method (conda, pip, source): source

<!-- If you are reporting an issue such as scale stability, cluster deadlock.
Please provide a cluster dump state with this issue, by running client.dump_cluster_state()

https://distributed.dask.org/en/stable/api.html?highlight=dump_cluster_state#distributed.Client.dump_cluster_state

-->

<details>
<summary>Cluster Dump State:</summary>

</details>


Discussion:
Thanks @phobson @ncclementi, I'm able to reproduce the dtype mismatch issue. Experimenting a bit locally, I was able to reproduce the same issue with a slightly simpler example that just divides two Dask Series (so avoiding `groupby` altogether):

```python
import pandas as pd
import dask.dataframe as dd

# Ratio of two pandas Series
x = pd.Series([1, 2, 3], index=[True, False, False])
y = pd.Series([4, 5, 6], index=[False, True, False])
z = x / y

# Dask version
dx = dd.from_pandas(x, npartitions=2)
dy = dd.from_pandas(y, npartitions=2)
dz = dx / dy

assert z.index.dtype is dz.index.compute().dtype
assert z.index.dtype is dz.index.dtype, f"{repr(z.index.dtype)} is not {repr(dz.index.dtype)}"
```

Interestingly, if we actually `compute()` the Dask Index, then we get the same dtype as what we would expect from pandas (i.e. `assert z.index.dtype is dz.index.compute().dtype`). This makes me think the issue is an issue with our DataFrame `meta` inference. The next place I'd look would be our Series `truediv` implementation (that should be the method that gets called when we use the `/` operator)

Thanks for the direction, @jrbourbeau. I'll coordinate with Naty.

Resolving patch:
diff --git a/dask/dataframe/backends.py b/dask/dataframe/backends.py
--- a/dask/dataframe/backends.py
+++ b/dask/dataframe/backends.py
@@ -194,7 +194,13 @@ def _nonempty_index(idx):
     elif idx.is_numeric():
         return typ([1, 2], name=idx.name)
     elif typ is pd.Index:
-        return pd.Index(["a", "b"], name=idx.name)
+        if idx.dtype == bool:
+            # pd 1.5 introduce bool dtypes and respect non-uniqueness
+            return pd.Index([True, False], name=idx.name)
+        else:
+            # for pd 1.5 in the case of bool index this would be cast as [True, True]
+            # breaking uniqueness
+            return pd.Index(["a", "b"], name=idx.name, dtype=idx.dtype)
     elif typ is pd.DatetimeIndex:
         start = "1970-01-01"
         # Need a non-monotonic decreasing index to avoid issues with


Test patch:
diff --git a/dask/dataframe/tests/test_groupby.py b/dask/dataframe/tests/test_groupby.py
--- a/dask/dataframe/tests/test_groupby.py
+++ b/dask/dataframe/tests/test_groupby.py
@@ -10,7 +10,7 @@
 import dask
 import dask.dataframe as dd
 from dask.dataframe import _compat
-from dask.dataframe._compat import PANDAS_GT_110, tm
+from dask.dataframe._compat import PANDAS_GT_110, PANDAS_GT_150, tm
 from dask.dataframe.utils import assert_dask_graph, assert_eq, assert_max_deps
 from dask.utils import M
 
@@ -1188,7 +1188,8 @@ def call(g, m, **kwargs):
         pytest.param(
             lambda df: [df["a"] > 2, df["b"] > 1],
             marks=pytest.mark.xfail(
-                reason="index dtype does not coincide: boolean != empty"
+                not PANDAS_GT_150,
+                reason="index dtype does not coincide: boolean != empty",
             ),
         ),
     ],