Repository: NVIDIA-Merlin/NVTabular

Issue:
[OP] DifferenceLag
**Is your feature request related to a problem? Please describe.**
DifferenceLag calculates the difference between the current row value and the previous/next row value. 

**Describe the solution you'd like**
DifferenceLag Logic is following:
1. sorts the data frame given a key column
2. shifts the to transformed column n steps (positive n shifts forward, negative n shits backward
3. calculates the difference between the current value and the shifted value

DifferenceLag can be applied with a category column - meaning that differences are only calculated with the current value and shifted value have the same category. Kind of groupby version.

**Additional context**
Implementation examples:
https://github.com/rapidsai/recsysChallenge2020/blob/master/Baselines/DASK-XGB/Demo-e2e-multi-CPU.ipynb : diff_encode_cudf_v1
https://github.com/rapidsai/recsysChallenge2020/blob/master/final_models/XGB1334/XGB1334-submit-singleGPU.ipynb : diff_encode_cudf_v1


Discussion:

Resolving patch:
diff --git a/nvtabular/ops.py b/nvtabular/ops.py
--- a/nvtabular/ops.py
+++ b/nvtabular/ops.py
@@ -1600,3 +1600,61 @@ def apply_op(
         new_gdf = self.f(gdf)
         new_gdf.reset_index(drop=True, inplace=True)
         return new_gdf
+
+
+class DifferenceLag(TransformOperator):
+    """ Calculates the difference between two consecutive rows of the dataset. For instance, this
+    operator can calculate the time since a user last had another interaction.
+
+    This requires a dataset partitioned by one set of columns (userid) and sorted further by another
+    set (userid, timestamp). The dataset must already be partitioned and sorted before being passed
+    to the workflow. This can be easily done using dask-cudf::
+
+        # get a nvt dataset and convert to a dask dataframe
+        ddf = nvtabular.Dataset(PATHS).to_ddf()
+
+        # partition the dask dataframe by userid, then sort by userid/timestamp
+        ddf = ddf.shuffle("userid").sort_values(["userid", "timestamp"])
+
+        # create a new nvtabular dataset on the partitioned/sorted values
+        dataset = nvtabular.Dataset(ddf)
+
+    Once passed an appropiate dataset, this operator can be added to a nvtabular workflow to
+    compute the lagged difference within a partition::
+
+        # compute the delta in timestamp for each users session
+        workflow.add_feature(DifferenceLag("userid', columns=["timestamp"]))
+
+    Parameters
+    -----------
+    partition_cols : str or list of str
+        Column or Columns that are used to partition the data.
+    shift : int, default 1
+        The number of rows to look backwards when computing the difference lag. Negative values
+        indicate the number of rows to look forwards, making this compute the lead instead of lag.
+    columns :
+    replace: bool, default False
+        Whether to replace existing columns or create new ones
+    """
+
+    default_in = CONT
+    default_out = CONT
+
+    def __init__(self, partition_cols, shift=1, columns=None, replace=False):
+        super(DifferenceLag, self).__init__(columns=columns, replace=replace)
+        self.partition_cols = partition_cols
+        self.shift = shift
+
+    @annotate("DifferenceLag_op", color="darkgreen", domain="nvt_python")
+    def op_logic(self, gdf: cudf.DataFrame, target_columns: list, stats_context=None):
+        # compute a mask indicating partition boundaries, handling multiple partition_cols
+        # represent partition boundaries by None values
+        mask = gdf[self.partition_cols] == gdf[self.partition_cols].shift(self.shift)
+        if isinstance(mask, cudf.DataFrame):
+            mask = mask.all(axis=1)
+        mask[mask == False] = None  # noqa
+
+        output = {}
+        for col in target_columns:
+            output[f"{col}_{self._id}"] = (gdf[col] - gdf[col].shift(self.shift)) * mask
+        return cudf.DataFrame(output)


Test patch:
diff --git a/tests/unit/test_ops.py b/tests/unit/test_ops.py
--- a/tests/unit/test_ops.py
+++ b/tests/unit/test_ops.py
@@ -876,3 +876,22 @@ def test_filter(tmpdir, df, dataset, gpu_memory_frac, engine, client):
     new_gdf = filter_op.apply_op(df, columns_ctx, "all", target_cols=columns)
     assert new_gdf.columns.all() == df.columns.all()
     assert new_gdf.shape[0] < df.shape[0], "null values do not exist"
+
+
+def test_difference_lag():
+    df = cudf.DataFrame(
+        {"userid": [0, 0, 0, 1, 1, 2], "timestamp": [1000, 1005, 1100, 2000, 2001, 3000]}
+    )
+
+    columns = ["userid", "timestamp"]
+    columns_ctx = {}
+    columns_ctx["all"] = {}
+    columns_ctx["all"]["base"] = columns
+
+    op = ops.DifferenceLag("userid", columns=["timestamp"])
+    new_gdf = op.apply_op(df, columns_ctx, "all", target_cols=["timestamp"])
+
+    assert new_gdf["timestamp_DifferenceLag"][0] is None
+    assert new_gdf["timestamp_DifferenceLag"][1] == 5
+    assert new_gdf["timestamp_DifferenceLag"][2] == 95
+    assert new_gdf["timestamp_DifferenceLag"][3] is None