Repository: pandas-dev/pandas

Issue:
corrupted data and segfault in groupby cumsum/cumprod/cummin/cummax with absent categories
#### Code Sample, a copy-pastable example if possible

Requires the contents of [bug.pkl.zip](https://github.com/pandas-dev/pandas/files/1102273/bug.pkl.zip)

```python
>>> import pandas as pd
>>> pd.read_pickle('bug.pkl').groupby('x').y.cummax()
*** Error in `python': double free or corruption (out): 0x00000000035784c0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7570b)[0x7f86d533f70b]
/lib64/libc.so.6(+0x7deaa)[0x7f86d5347eaa]
/lib64/libc.so.6(cfree+0x4c)[0x7f86d534b40c]
/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so(+0x5c2ed)[0x7f86ce5f12ed]
/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so(+0x2013e)[0x7f86ce5b513e]
/lib/python3.6/site-packages/pandas/_libs/groupby.cpython-36m-x86_64-linux-gnu.so(+0x69a76)[0x7f86bb8bba76]
/lib/python3.6/site-packages/pandas/_libs/groupby.cpython-36m-x86_64-linux-gnu.so(+0x6ad39)[0x7f86bb8bcd39]
...
```
#### Problem description

Calling `groupby().cummax()` on the attached dataframe in a new python process results in a segfault on my machine. Not sure why this dataframe specifically; I couldn't find a simple test case that caused this. Not sure if it's a numpy or pandas issue, or how machine-specific it is.

#### Expected Output

Not a segfault.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.10.10-100.fc24.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C
LANG: C
LOCALE: None.None

pandas: 0.20.2
pytest: 3.1.2
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.0
scipy: 0.19.0
xarray: None
IPython: 4.2.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.2
feather: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
This appears to be a regression in v0.20, when the `groupby().cummax()` implementation moved to cython. Storing the dataframe in HDF5 for compatibility across pandas versions, I can reproduce the same issue in v0.20.x, but not in v0.19.2.
pls show the actual frame with construction and df.info() exactly as it looks

even if it doesn't repro the issue
See below for a simpler test-case. Issue appears to be caused when grouping by categories that don't appear in the frame.

```python
import pandas as pd
import numpy as np

x_vals = np.arange(2) + 2
x_cats = np.arange(4)
y = np.arange(2.0)
df = pd.DataFrame(dict(x=pd.Categorical(x_vals, x_cats), y=y))

df.groupby('x').y.cummax()
```

```python
>>> df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
x    2 non-null category
y    2 non-null float64
dtypes: category(1), float64(1)
memory usage: 290.0 bytes
```

yeah shouldn't segfault but this is also not implemented (shouldn't be hard though)
note the same test case also fails on `cumsum()` -- the problem might affect all grouped-cumulative operations
Works on master now. Could use a test

```
In [4]: pd.__version__
Out[4]: '0.25.0.dev0+85.g0eddba883'

In [5]: import pandas as pd
   ...: import numpy as np
   ...:
   ...: x_vals = np.arange(2) + 2
   ...: x_cats = np.arange(4)
   ...: y = np.arange(2.0)
   ...: df = pd.DataFrame(dict(x=pd.Categorical(x_vals, x_cats), y=y))
   ...:
   ...: df.groupby('x').y.cumsum()
Out[5]:
0    1.730601e-77
1    1.000000e+00
Name: y, dtype: float64
```
@mroeschke 
When I am running your code, it runs without segfault, but when the process (python interpreter or through script) ends, it ends with a seg fault. Even the original code segment with ```bug.pkl``` gives a seg fault when executed as

```
import pandas as pd
a = pd.read_pickle('bug.pkl')
a.groupby('x').y.cummax()
```

My pandas version is 0.24.0. I'll once try the master version tonight and see if the issue persists.

So I ran the above code in the current master. The pandas version is
```
>>> pd.__version__
'0.25.0.dev0+113.gb8306f19d'
```
I got the following error after the interpreter crashed
```
invalid fastbin entry (free)
Aborted
```

Which interpreter are you using? Works fine in ipython

```
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 14:01:38)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: pd.__version__
Out[1]: '0.25.0.dev0+118.gb4913dc92'

In [2]: In [5]: import pandas as pd
   ...:    ...: import numpy as np
   ...:    ...:
   ...:    ...: x_vals = np.arange(2) + 2
   ...:    ...: x_cats = np.arange(4)
   ...:    ...: y = np.arange(2.0)
   ...:    ...: df = pd.DataFrame(dict(x=pd.Categorical(x_vals, x_cats), y=y))
   ...:    ...:
   ...:    ...: df.groupby('x').y.cumsum()
Out[2]:
0    9.881313e-324
1     1.000000e+00
Name: y, dtype: float64
```
Here's the complete thing.

```
$ python
Python 3.7.2 (default, Dec 29 2018, 06:19:36) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import pandas as pd
>>> a = pd.read_pickle('~/Downloads/bug.pkl')
>>> a.groupby('x').y.cummax()
invalid fastbin entry (free)
Aborted
```

Same thing when I tried your code

```
$ python
Python 3.7.2 (default, Dec 29 2018, 06:19:36) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import pandas as pd
>>> import numpy as np
>>> 
>>> x_vals = np.arange(2) + 2
>>> x_cats = np.arange(4)
>>> y = np.arange(2.0)
>>> df = pd.DataFrame(dict(x=pd.Categorical(x_vals, x_cats), y=y))
>>> 
>>> df.groupby('x').y.cumsum()
invalid fastbin entry (free)
Aborted
```

Maybe a 3.6 vs 3.7 issue? Could you create a 3.6 environment and see whether this still segfaults?
3.6 gives a similar issue. Different segfault errors I got-

```
invalid fastbin entry (free)
aborted
```
or
```
malloc(): memory corruption
aborted
```
or
```
malloc_consolidate(): invalid chunk size
Aborted
```
or
```
corrupted size vs. prev_size
Aborted
```

Python version
```
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 12:22:00) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
```

Okay this may still be segfaulting on Linux then (I'm running OSX).
Any idea which parts of the source code I should look into, for the issue?
https://github.com/pandas-dev/pandas/blob/4be995c13452766da48a82f3ecd0b5a24e48f526/pandas/_libs/groupby_helper.pxi.in#L522

https://github.com/pandas-dev/pandas/blob/4be995c13452766da48a82f3ecd0b5a24e48f526/pandas/_libs/groupby_helper.pxi.in#L474
I don't understand what change I have to make here.

1. The code doesn't segfault everytime.
2. ```values``` and ```labels``` are only getting what they should be getting *ie* data without the missing categories. What check should I write here?
The issue is caused by various cython helper functions incorrectly allocating temporary arrays with shape `(rows, columns)`, when they should have shape `(groups, columns)`. This fails when `groups > rows`, which is possible for categorical groupers with unused categories.

The bug can also cause corrupted data rather than segfaulting, e.g. as below:
```python
import pandas as pd

x_vals = [1]
x_cats = range(2)
y = [1]
df = pd.DataFrame(dict(x=pd.Categorical(x_vals, x_cats), y=y))

print(df.groupby('x').y.cumsum())
```
This should print 1, but instead it prints a random value, at least on linux. 

Resolving patch:
diff --git a/pandas/core/groupby/ops.py b/pandas/core/groupby/ops.py
--- a/pandas/core/groupby/ops.py
+++ b/pandas/core/groupby/ops.py
@@ -361,8 +361,8 @@ def get_group_levels(self):
             'cummax': 'group_cummax',
             'rank': {
                 'name': 'group_rank',
-                'f': lambda func, a, b, c, d, **kwargs: func(
-                    a, b, c, d,
+                'f': lambda func, a, b, c, d, e, **kwargs: func(
+                    a, b, c, e,
                     kwargs.get('ties_method', 'average'),
                     kwargs.get('ascending', True),
                     kwargs.get('pct', False),
@@ -600,9 +600,10 @@ def _transform(self, result, values, comp_ids, transform_func,
             for i, chunk in enumerate(values.transpose(2, 0, 1)):
 
                 transform_func(result[:, :, i], values,
-                               comp_ids, is_datetimelike, **kwargs)
+                               comp_ids, ngroups, is_datetimelike, **kwargs)
         else:
-            transform_func(result, values, comp_ids, is_datetimelike, **kwargs)
+            transform_func(result, values, comp_ids, ngroups, is_datetimelike,
+                           **kwargs)
 
         return result
 


Test patch:
diff --git a/pandas/tests/groupby/test_transform.py b/pandas/tests/groupby/test_transform.py
--- a/pandas/tests/groupby/test_transform.py
+++ b/pandas/tests/groupby/test_transform.py
@@ -9,7 +9,8 @@
 from pandas.core.dtypes.common import ensure_platform_int, is_timedelta64_dtype
 
 import pandas as pd
-from pandas import DataFrame, MultiIndex, Series, Timestamp, concat, date_range
+from pandas import (
+    Categorical, DataFrame, MultiIndex, Series, Timestamp, concat, date_range)
 from pandas.core.groupby.groupby import DataError
 from pandas.util import testing as tm
 from pandas.util.testing import assert_frame_equal, assert_series_equal
@@ -470,7 +471,8 @@ def _check_cython_group_transform_cumulative(pd_op, np_op, dtype):
     ans = np.zeros_like(data)
 
     labels = np.array([0, 0, 0, 0], dtype=np.int64)
-    pd_op(ans, data, labels, is_datetimelike)
+    ngroups = 1
+    pd_op(ans, data, labels, ngroups, is_datetimelike)
 
     tm.assert_numpy_array_equal(np_op(data), ans[:, 0],
                                 check_dtype=False)
@@ -496,17 +498,19 @@ def test_cython_group_transform_algos():
 
     # with nans
     labels = np.array([0, 0, 0, 0, 0], dtype=np.int64)
+    ngroups = 1
 
     data = np.array([[1], [2], [3], [np.nan], [4]], dtype='float64')
     actual = np.zeros_like(data)
     actual.fill(np.nan)
-    groupby.group_cumprod_float64(actual, data, labels, is_datetimelike)
+    groupby.group_cumprod_float64(actual, data, labels, ngroups,
+                                  is_datetimelike)
     expected = np.array([1, 2, 6, np.nan, 24], dtype='float64')
     tm.assert_numpy_array_equal(actual[:, 0], expected)
 
     actual = np.zeros_like(data)
     actual.fill(np.nan)
-    groupby.group_cumsum(actual, data, labels, is_datetimelike)
+    groupby.group_cumsum(actual, data, labels, ngroups, is_datetimelike)
     expected = np.array([1, 3, 6, np.nan, 10], dtype='float64')
     tm.assert_numpy_array_equal(actual[:, 0], expected)
 
@@ -515,7 +519,7 @@ def test_cython_group_transform_algos():
     data = np.array([np.timedelta64(1, 'ns')] * 5, dtype='m8[ns]')[:, None]
     actual = np.zeros_like(data, dtype='int64')
     groupby.group_cumsum(actual, data.view('int64'), labels,
-                         is_datetimelike)
+                         ngroups, is_datetimelike)
     expected = np.array([np.timedelta64(1, 'ns'), np.timedelta64(
         2, 'ns'), np.timedelta64(3, 'ns'), np.timedelta64(4, 'ns'),
         np.timedelta64(5, 'ns')])
@@ -863,3 +867,16 @@ def test_groupby_transform_with_datetimes(func, values):
                          index=dates, name="price")
 
     tm.assert_series_equal(result, expected)
+
+
+@pytest.mark.parametrize('func', ['cumsum', 'cumprod', 'cummin', 'cummax'])
+def test_transform_absent_categories(func):
+    # GH 16771
+    # cython transforms with more groups than rows
+    x_vals = [1]
+    x_cats = range(2)
+    y = [1]
+    df = DataFrame(dict(x=Categorical(x_vals, x_cats), y=y))
+    result = getattr(df.y.groupby(df.x), func)()
+    expected = df.y
+    assert_series_equal(result, expected)