Repository: dask/dask

Issue:
[DISCUSSION] Remove pandas groupby.last() dependency from _take_last method
Currently, cumulative aggregation operations are tied to non-hash based groupbys and pandas through the process of grabbing the last row and explicit pandas mentions in several places, particularly in [_take_last](https://github.com/dask/dask/blob/master/dask/dataframe/core.py#L4179).

Handling the explicit pandas references is already a TODO, but I'd like to raise for discussion removing the usage of  `groupby.last()`. Using pd.DataFrame.groupby.last() with a dummy group to capture the last row of each column works because pandas groupby preserves the order of rows within each group. Hash-based cuDF groupbys do not make such a guarantees, making this approach incompatible with cuDF.

There are several potential ways to change this if there is interest, and they have varying performance penalties / cleanliness. All of the approaches proposed below have a performance penalty that increases with the number of columns.

The clearest ways are to either apply `pd.Series.last_valid_index` and then perform the selection via the underlying numpy array, or to essentially mimic the functionality of `last_valid_index` either explicitly or in a large lambda function expression.

Given the following example data, here are some loose time benchmarks of approaches to getting the same data:
```
import pandas as pd
nrows = 1000000 # one million rows
pdf = pd.DataFrame({'a':['abc']*(nrows),
                     'b':range(nrows),
                     'c':range(nrows),
                     'd':range(nrows)
                    })
```

Using the current approach appears to be the fastest, but causes the issues listed above:

```
%%time
group_dummy = np.ones(len(pdf.index))
print(pdf.groupby(group_dummy).last())
       a       b       c       d
1.0  abc  999999  999999  999999
CPU times: user 80 ms, sys: 4 ms, total: 84 ms
Wall time: 85.7 ms
```

Explicitly mimicking pandas's `groupby.last` method (with some slight changes to remove explicitly using the underlying numpy array) found [here](https://github.com/pandas-dev/pandas/blob/v0.24.2/pandas/core/groupby/groupby.py#L1288)
```
def last(x):
    x = x[x.notna()]
    if len(x) == 0:
        return np.nan
    return x.iloc[-1]

pd.DataFrame(pdf.apply(last)).T
     a       b       c       d
0  abc  999999  999999  999999
CPU times: user 312 ms, sys: 40 ms, total: 352 ms
Wall time: 352 ms
```

Using a large lambda expression that functionally does the same as using `last`:

```
%time print(pd.DataFrame(pdf.apply(lambda x: x[x.notna()].iloc[-1] if not x.isnull().all() else None)).T)
     a       b       c       d
0  abc  999999  999999  999999
CPU times: user 452 ms, sys: 32 ms, total: 484 ms
Wall time: 484 ms
```

Using `last_valid_index`:

```
%%time
ix = pdf.apply(pd.Series.last_valid_index)
print(pd.DataFrame(pdf.values[ix, np.arange(pdf.shape[1])],index=df.columns).T)
     a       b       c       d
0  abc  999999  999999  999999
CPU times: user 372 ms, sys: 88 ms, total: 460 ms
Wall time: 456 ms
```
I wasn't able to easily figure out a good way to avoid using numpy with `last_valid_index`, but it's likely feasible. If we can't do that, this approach does not resolve the issues in a way that GPU libraries like cuDF can leverage.

In general, all of these non-`groupby.last() ` based approaches have a performance penalty of about 4-5x at 1 million records (example above) of the dataset above, 10 million records (1 second vs 4-5 seconds), and 50 million records (5 seconds vs 20-25 seconds).

**The problem though, is that the performance penalty increases with the number of columns unfortunately**. So if we change the example above to have seven columns, at 10 million records the groupby approach is still ~1 second but the non-groupby approach takes ~7 seconds.


I'd like to hear people's thoughts on this topic, as we'd like to be able to support dask-based cumulative aggregations for GPU dataframes. Do people have a sense of whether this kind of performance tradeoff is reasonable?


[BUG] cumulative aggregations fail when there are empty partitions
### Description
As @mrocklin noticed, the cumulative aggregation functions will error when there are empty partitions, due to the `_take_last` function hitting an index error trying to grab the 0th element of an empty array.

### Example

```python
import pandas as pd
import dask.dataframe as dd
df = pd.DataFrame({'x': [1, 2, 3, 4, 5, 6, 7, 8]})
ddf = dd.from_pandas(df, npartitions=4)
ddf[ddf.x < 5].cumsum().compute()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-5-f2ddb62cae63> in <module>()
----> 1 ddf[ddf.x < 5].cumsum().compute()

/Users/nickbecker/NVIDIA/dask/dask/base.pyc in compute(self, **kwargs)
    154         dask.base.compute
    155         """
--> 156         (result,) = compute(self, traverse=False, **kwargs)
    157         return result
    158 

/Users/nickbecker/NVIDIA/dask/dask/base.pyc in compute(*args, **kwargs)
    396     keys = [x.__dask_keys__() for x in collections]
    397     postcomputes = [x.__dask_postcompute__() for x in collections]
--> 398     results = schedule(dsk, keys, **kwargs)
    399     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    400 

/Users/nickbecker/NVIDIA/dask/dask/threaded.pyc in get(dsk, result, cache, num_workers, pool, **kwargs)
     74     results = get_async(pool.apply_async, len(pool._pool), dsk, result,
     75                         cache=cache, get_id=_thread_get_id,
---> 76                         pack_exception=pack_exception, **kwargs)
     77 
     78     # Cleanup pools associated to dead threads

/Users/nickbecker/NVIDIA/dask/dask/local.pyc in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs)
    460                         _execute_task(task, data)  # Re-execute locally
    461                     else:
--> 462                         raise_exception(exc, tb)
    463                 res, worker_id = loads(res_info)
    464                 state['cache'][key] = res

/Users/nickbecker/NVIDIA/dask/dask/local.pyc in execute_task(key, task_info, dumps, loads, get_id, pack_exception)
    228     try:
    229         task, data = loads(task_info)
--> 230         result = _execute_task(task, data)
    231         id = get_id()
    232         result = dumps((result, id))

/Users/nickbecker/NVIDIA/dask/dask/core.pyc in _execute_task(arg, cache, dsk)
    117         func, args = arg[0], arg[1:]
    118         args2 = [_execute_task(a, cache) for a in args]
--> 119         return func(*args2)
    120     elif not ishashable(arg):
    121         return arg

/Users/nickbecker/NVIDIA/dask/dask/optimization.pyc in __call__(self, *args)
    940                              % (len(self.inkeys), len(args)))
    941         return core.get(self.dsk, self.outkey,
--> 942                         dict(zip(self.inkeys, args)))
    943 
    944     def __reduce__(self):

/Users/nickbecker/NVIDIA/dask/dask/core.pyc in get(dsk, out, cache)
    147     for key in toposort(dsk):
    148         task = dsk[key]
--> 149         result = _execute_task(task, cache)
    150         cache[key] = result
    151     result = _execute_task(out, cache)

/Users/nickbecker/NVIDIA/dask/dask/core.pyc in _execute_task(arg, cache, dsk)
    117         func, args = arg[0], arg[1:]
    118         args2 = [_execute_task(a, cache) for a in args]
--> 119         return func(*args2)
    120     elif not ishashable(arg):
    121         return arg

/Users/nickbecker/NVIDIA/dask/dask/dataframe/core.py in apply_and_enforce(*args, **kwargs)
   3869     func = kwargs.pop('_func')
   3870     meta = kwargs.pop('_meta')
-> 3871     df = func(*args, **kwargs)
   3872     if is_dataframe_like(df) or is_series_like(df) or is_index_like(df):
   3873         if not len(df):

/Users/nickbecker/NVIDIA/dask/dask/dataframe/core.py in _take_last(a, skipna)
   4213         last_row = a.groupby(group_dummy).last()
   4214         if isinstance(a, pd.DataFrame):  # TODO: handle explicit pandas reference
-> 4215             return pd.Series(last_row.values[0], index=a.columns)
   4216         else:
   4217             return last_row.values[0]

IndexError: index 0 is out of bounds for axis 0 with size 0
```


Discussion:
What about a Python for loop that walks from the end of each series to the front that checks for a non-null value?

```python
out = {}
for c in df.columns:
    for i in range(...):
        value = df[c].iloc[-i]
        if not_null(value):
            out[c] = value
            break
...

In the worst case scenario (all null) this is probably more expensive than the approaches listed above, but in the common case I suspect that it is far faster.  
That would certainly work. I'd defer to you and others as to how common that the scenario of having to iterate over many columns most of the way up would be, as I think the penalty could be significant if so.
```python
In [1]: import pandas as pd
   ...: nrows = 1000000 # one million rows
   ...: pdf = pd.DataFrame({'a':['abc']*(nrows),
   ...:                      'b':range(nrows),
   ...:                      'c':range(nrows),
   ...:                      'd':range(nrows)
   ...:                     })

In [2]: %timeit s = pdf['b']
1.2 µs ± 7.94 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [3]: s = pdf['b']

In [4]: %timeit s.iloc[-1]
5.39 µs ± 46 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

So if we had 1000 columns (which I'm going to claim as extreme), each of which had no null elements, this approach would take 1ms + 6ms, which is fine.  There are some other challenges I think:

1.  In the case where there are a ton of null elements (thousands before we got to a non-null element) we would be incurring something like 6ms per column (which is probably ok still)
2.  It's not clear to me how to test for nullability on the resulting values.  
3.  I wouldn't be surprised if cudf's iloc is significantly slower than this, you should probably check that out and see if it's running efficiently.

My guess though, is that there is probably some more clever way to do all of this.  This is one of those cases where it'd be really nice to have someone with a lot of Pandas knowledge around.  Unfortunately Dask's resident Panda is out for a couple more weeks on family leave.

From my perspective and thinking about the distribution of cases I see with Pandas, the Python for loop solution seems fine to me if we can address point 2 above.  I'm particularly concerned about exotic types like `Int64` (Pandas nullable int type) or datetimes, which may have other kinds of null checks.
cudf's iloc is indeed slower (about 20x in the cudf version of the above example).

Here is a pandas example with some loose timing for the worst case scenario:

100,000 rows, 10 columns
```python
def last_valid_val(df): 
    out = {}
    for col in df.columns:
        for i in range(df.shape[0]-1, -1, -1):
            val = df[col].iloc[i]
            if not pd.isnull(val):
                out[col] = val
                break
    return pd.DataFrame.from_dict(out, orient='index').T


pdf = pd.DataFrame()
ncols = 10
nrows = 100000
for i in range(ncols):
    pdf['col_{}'.format(i)] = [i] + [None]*nrows

%timeit o = last_valid_val(pdf)
9.99 s ± 30.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```
Null checking every value independently is the largest timesink. Since it takes ~10-20 us per nan/nullcheck (and there may be a faster way of course), then at 100,000 values per column it takes 1-2 seconds per column to handle the null checking.

Regarding point 2 above, the existing implementation is leveraging numpy's `notna` [under the hood](https://github.com/pandas-dev/pandas/blob/v0.24.2/pandas/core/groupby/groupby.py#L1290) to filter out the non-null values. If the resulting array is empty, it outputs np.nan. Could we do the same logic in terms of null checking and leverage pandas/cudf/other future dataframe library's `notna/notnull` method? Going briefly down the (long) rabbit hole of pandas notna in the past 10 minutes, it appears to be leveraging numpy's notna and some custom cython around it in many situations.

If we wanted to get crazy in terms of performance, we could start with the for loop solution for maybe 10 iterations, and then if that didn't work, we could switch to one of the bulk solutions that you suggest above?

```python
    out = {}
    for col in df.columns:
        s = df[col]
        for i in range(0, min(10, len(df)):
            val = s.iloc[-i]
            if not pd.isnull(val):
                out[col] = val
                break
        else:  # ooh, a reason for the rarely used for-else construct :)
            nonnull = s.notnull()
            if len(nonull):
                out[col] = nonnull.iloc[-1]
            else:
                ... 
```
I'm generally a fan of that kind of camel solution if it provides the value to end users and doesn't add too much complexity/confusing code for future development. I'd be happy to make a such change combining these ideas and verify if tests still pass.
I'm in favor of it.  You might also want to check on the cudf side to verify that `notnull` is implemented (and if not, hopefully it's an easy fix).
That kind of null support is not yet implemented, but will be eventually (some of it is in flight currently). I'd initially try test this change with a methodcaller version of all pandas functions such that it could eventually automatically support the analogous cudf methods.
Yeah, so in the code above I never actually call out pandas, I just use methods, so hopefully it would work as-is.
Combining the logic works beautifully performance wise (in pandas at least), and should succeed when cuDF implements `isnull` and `notna` at the Series level.

One potential question I'd like to discuss here given the use of `pd.isnull(val)` is how to best handle the usage of top-level functions. We can operate on the series by slicing instead of direct indexing above, and then leverage the `isnull` method through the methodcaller. If we add the top level pandas module `pd` to the `_method_cache`, we can use top level functions as if they were methods. This does likely bloat things, though. Figured I'd raise this point here to possibly learn more about it the pros/cons of this in general.

```
...
M = methodcaller(pd)
M.method.isnull(5)
```

The **overall** performance impact of using the series method `isnull` is fairly small all things considered (using the combined logic above), so I think this is the best solution still. Grabbing the last valid value of each column in a dataframe with 50 columns and 10 million rows takes < 1 second. Both approaches are **far** faster than the current groupby.last() approach at this scale, too, and comparable at smaller scales.

We're currently only using `pd.isnull` on scalar values.  I suspect that this won't conflict with cudf.

I'm not sure I understand the value of wrapping pandas in methodcaller.  That seems to be the same as just using `pd`.
Good point. I had in mind a potential scenario in which we would want to use the top level function of a specific module (pandas or cudf) depending on the data structure and was curious about that as a possibility, but you're absolutely right. It's not necessary here due to the scalar.


Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -4204,17 +4204,31 @@ def _take_last(a, skipna=True):
         Whether to exclude NaN
 
     """
+    def _last_valid(s):
+        for i in range(1, min(10, len(s) + 1)):
+            val = s.iloc[-i]
+            if not pd.isnull(val):
+                return val
+        else:
+            nonnull = s[s.notna()]
+            if not nonnull.empty:
+                return nonnull.iloc[-1]
+        return None
+
     if skipna is False:
         return a.iloc[-1]
     else:
         # take last valid value excluding NaN, NaN location may be different
-        # in each columns
-        group_dummy = np.ones(len(a.index))
-        last_row = a.groupby(group_dummy).last()
-        if isinstance(a, pd.DataFrame):  # TODO: handle explicit pandas reference
-            return pd.Series(last_row.values[0], index=a.columns)
+        # in each column
+        if is_dataframe_like(a):
+            # create Series from appropriate backend dataframe library
+            series_typ = type(a.loc[0:1, a.columns[0]])
+            if a.empty:
+                return series_typ([])
+            return series_typ({col: _last_valid(a[col]) for col in a.columns},
+                              index=a.columns)
         else:
-            return last_row.values[0]
+            return _last_valid(a)
 
 
 def check_divisions(divisions):


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -405,6 +405,25 @@ def test_cumulative():
     assert_eq(df.cumprod(axis=1, skipna=False), ddf.cumprod(axis=1, skipna=False))
 
 
+@pytest.mark.parametrize(
+    'func',
+    [M.cumsum,
+     M.cumprod,
+     pytest.param(M.cummin, marks=[pytest.mark.xfail(
+         reason='ValueError: Can only compare identically-labeled Series objects')]),
+     pytest.param(M.cummax, marks=[pytest.mark.xfail(
+         reason='ValueError: Can only compare identically-labeled Series objects')])]
+)
+def test_cumulative_empty_partitions(func):
+    df = pd.DataFrame({'x': [1, 2, 3, 4, 5, 6, 7, 8]})
+    ddf = dd.from_pandas(df, npartitions=4)
+    assert_eq(func(df[df.x < 5]), func(ddf[ddf.x < 5]))
+
+    df = pd.DataFrame({'x': [1, 2, 3, 4, None, 5, 6, None, 7, 8]})
+    ddf = dd.from_pandas(df, npartitions=5)
+    assert_eq(func(df[df.x < 5]), func(ddf[ddf.x < 5]))
+
+
 def test_dropna():
     df = pd.DataFrame({'x': [np.nan, 2, 3, 4, np.nan, 6],
                        'y': [1, 2, np.nan, 4, np.nan, np.nan],