Repository: dask/dask

Issue:
Generalizing rolling window functionality to dataframe backends
Similar to #5133 , we should generalize the rolling window functionality. Currently, there is explicit concatenation inside `overlap_chunks` that ties rolling windows to pandas, though there may be other areas as well (I'm investigating). That there is a reference to `pd.foo` does not inherently cause a generalization issue.

https://github.com/dask/dask/blob/d0ee3203ea0c65eeb5effb2efe1c4544b9317e4b/dask/dataframe/rolling.py#L34-L36

This issue can serve as a tracker for rolling window related generalization.

- [x] dask.dataframe.rolling.overlap_chunk
- [x] dask.dataframe.rolling._tail_timedelta


Discussion:
We already have a generalized concat function used in various other parts
of the code.  My guess is that we can swap that in relatively painlessly.

On Mon, Jul 22, 2019 at 3:32 PM Nick Becker <notifications@github.com>
wrote:

> Similar to #5133 <https://github.com/dask/dask/issues/5133> , we should
> generalize the rolling window functionality. Currently, there is explicit
> concatenation inside overlap_chunks that ties rolling windows to pandas,
> though there may be other areas as well (I'm investigating). That there is
> a reference to pd.foo does not inherently cause a generalization issue.
>
>
> https://github.com/dask/dask/blob/d0ee3203ea0c65eeb5effb2efe1c4544b9317e4b/dask/dataframe/rolling.py#L34-L36
>
> This issue can serve as a tracker for rolling window related
> generalization.
>
>    - dask.dataframe.rolling. overlap_chunk
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/5135?email_source=notifications&email_token=AACKZTD3CY6PFQCR4OWHRF3QAYYPTA5CNFSM4IF5RSQ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HAYQLYA>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACKZTBI5PC6226SWFITBADQAYYPTANCNFSM4IF5RSQQ>
> .
>

:thumbsup: @mrocklin . cc @galipremsagar . Any generalizations will be blocked in cuDF + Dask land by cosmetic changes we need to make on cuDF (rapidsai/cudf#2372)
Yes, looks like we use `methods.concat` for the dispatchable
(cudf-friendly) version.  My guess is that you could swap out `pd.concat`
with `methods.concat` and things would be fine.

On Mon, Jul 22, 2019 at 3:35 PM Nick Becker <notifications@github.com>
wrote:

> 👍 Matt. cc @galipremsagar <https://github.com/galipremsagar>
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/5135?email_source=notifications&email_token=AACKZTBURQCA77VUYKC7EGLQAYY3VA5CNFSM4IF5RSQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2RLSNY#issuecomment-513980727>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACKZTBAKF2UQOT2ADUD2FLQAYY3VANCNFSM4IF5RSQQ>
> .
>


Resolving patch:
diff --git a/dask/dataframe/rolling.py b/dask/dataframe/rolling.py
--- a/dask/dataframe/rolling.py
+++ b/dask/dataframe/rolling.py
@@ -11,6 +11,7 @@
 from ..highlevelgraph import HighLevelGraph
 from .core import _emulate
 from .utils import make_meta, PANDAS_VERSION
+from . import methods
 
 
 def overlap_chunk(
@@ -32,7 +33,7 @@ def overlap_chunk(
             raise NotImplementedError(msg)
 
     parts = [p for p in (prev_part, current_part, next_part) if p is not None]
-    combined = pd.concat(parts)
+    combined = methods.concat(parts)
     out = func(combined, *args, **kwargs)
     if prev_part is None:
         before = None
@@ -241,7 +242,7 @@ def _tail_timedelta(prevs, current, before):
     -------
     overlapped : DataFrame
     """
-    selected = pd.concat(
+    selected = methods.concat(
         [prev[prev.index > (current.index.min() - before)] for prev in prevs]
     )
     return selected


Test patch: