Repository: dask/dask (Python)

Issue:
GroupBy `std` aggregation raises `ValueError` 
When grouping by a key column that contains values that are above sqrt(max) for that column type, a `ValueError` can be thrown due to unique values overflowing into same-values, which invalidates the groupby results.

```
import pandas as pd
import numpy as np
import dask.dataframe as dd

max = np.iinfo(np.uint64).max
sqrt = max ** 0.5
series = pd.Series(np.concatenate([sqrt * np.arange(5), np.arange(35)])).astype(
    "int64"
)
df = pd.DataFrame({"x": series, "z": np.arange(40), "y": np.arange(40)})
ddf = dd.from_pandas(df, npartitions=1)
ddf.groupby("x").std().compute(scheduler="single-threaded")
```
raises
```
~/anaconda3/envs/cudf_dev/lib/python3.7/site-packages/pandas/core/internals/mana
gers.py in set_axis(self, axis, new_labels)
    153             raise ValueError(
    154                 'Length mismatch: Expected axis has {old} elements, new
'
--> 155                 'values have {new} elements'.format(old=old_len, new=new
_len))
    156
    157         self.axes[axis] = new_labels

ValueError: Length mismatch: Expected axis has 35 elements, new values have 39 e
lements
```

I've created #5737 to resolve this issue. My current solution is dumb - it uses `try` and `set` operations to only square the value columns, but not the `by` column/s. If that raises an exception, it reverts to the old method which assumes that a `Series` has been passed as the by key, which won't produce the problem. I'm open to suggestions for a smarter solution.


Discussion:

Patch:
diff --git a/dask/dataframe/groupby.py b/dask/dataframe/groupby.py
index 3c9165a3c07..e7592830cda 100644
--- a/dask/dataframe/groupby.py
+++ b/dask/dataframe/groupby.py
@@ -298,18 +298,18 @@ def _var_chunk(df, *index):
         df = df.to_frame()
 
     df = df.copy()
-    cols = df._get_numeric_data().columns
 
     g = _groupby_raise_unaligned(df, by=index)
     x = g.sum()
 
     n = g[x.columns].count().rename(columns=lambda c: (c, "-count"))
 
+    cols = x.columns
     df[cols] = df[cols] ** 2
+
     g2 = _groupby_raise_unaligned(df, by=index)
     x2 = g2.sum().rename(columns=lambda c: (c, "-x2"))
 
-    x2.index = x.index
     return concat([x, x2, n], axis=1)
 
 


Test patch:
diff --git a/dask/dataframe/tests/test_groupby.py b/dask/dataframe/tests/test_groupby.py
index 3ef7cf01798..cc4ba9a82fb 100644
--- a/dask/dataframe/tests/test_groupby.py
+++ b/dask/dataframe/tests/test_groupby.py
@@ -2329,3 +2329,24 @@ def test_groupby_split_out_multiindex(split_out, column):
     )
 
     assert_eq(ddf_result, ddf_result_so1, check_index=False)
+
+
+@pytest.mark.parametrize("backend", ["cudf", "pandas"])
+def test_groupby_large_ints_exception(backend):
+    data_source = pytest.importorskip(backend)
+    if backend == "cudf":
+        dask_cudf = pytest.importorskip("dask_cudf")
+        data_frame = dask_cudf.from_cudf
+    else:
+        data_frame = dd.from_pandas
+    max = np.iinfo(np.uint64).max
+    sqrt = max ** 0.5
+    series = data_source.Series(
+        np.concatenate([sqrt * np.arange(5), np.arange(35)])
+    ).astype("int64")
+    df = data_source.DataFrame({"x": series, "z": np.arange(40), "y": np.arange(40)})
+    ddf = data_frame(df, npartitions=1)
+    assert_eq(
+        df.groupby("x").std(),
+        ddf.groupby("x").std().compute(scheduler="single-threaded"),
+    )