Repository: NVIDIA-Merlin/NVTabular (language: python)

Issue:
[FEA] Create an op to drop unwanted column based on their `nunique` values
**Is your feature request related to a problem? Please describe.**

We are working on a dataset and  there are columns where their cardinality is 1. We want to drop these columns (vey common practice). Currently we use a function to do so:

```
def reduce_features(df):
    RMV = []
    for c in df.columns[1:]:
        if df[c].nunique()<=1:
            RMV.append(c)
    df.drop(RMV,axis=1,inplace=True)
    return
```
However, with large dataset we get OOM issue with cuDF so we have to use pandas instead which takes too long. It'd be useful to incorporate an OP in NVTabular.  

If there is a way to do it with NVT other than creating  a new op , I'd appreciate the feedback. Thanks.



Discussion / hints:
@rjzamora Hello. I created the FEA above. it is challenging to do that op since it requires calculation of `nunique()`? thanks.
I guess the "challenge" here is that the `fit` method for this op would be doing something very similar to `Categorify.fit` (perhaps the exact same thing), and I don't think there is any obvious way to avoid duplicating the work in NVTabular (yet). Therefore, if you are dropping columns based on unique AND using Categorify, you will be sampling unique values twice.  If you will **always** be calling `Categorify` on the columns that are not being dropped by your op, then I suppose you could just add a new option in `Categoify` to drop columns with too few uniques.

Fix patch:
diff --git a/nvtabular/ops/__init__.py b/nvtabular/ops/__init__.py
index ce7e08fa179..ed7f99c02bf 100644
--- a/nvtabular/ops/__init__.py
+++ b/nvtabular/ops/__init__.py
@@ -23,6 +23,7 @@
 from .column_similarity import ColumnSimilarity
 from .data_stats import DataStats
 from .difference_lag import DifferenceLag
+from .drop_low_cardinality import DropLowCardinality
 from .dropna import Dropna
 from .fill import FillMedian, FillMissing
 from .filter import Filter
diff --git a/nvtabular/ops/drop_low_cardinality.py b/nvtabular/ops/drop_low_cardinality.py
new file mode 100644
index 00000000000..8cf9894f89d
--- /dev/null
+++ b/nvtabular/ops/drop_low_cardinality.py
@@ -0,0 +1,47 @@
+#
+# Copyright (c) 2021, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from merlin.schema import Schema, Tags
+
+from ..dispatch import DataFrameType, annotate
+from .operator import ColumnSelector, Operator
+
+
+class DropLowCardinality(Operator):
+    """
+    DropLowCardinality drops low cardinality categorical columns. This requires the
+    cardinality of these columns to be known in the schema - for instance by
+    first encoding these columns using Categorify.
+    """
+
+    def __init__(self, min_cardinality=2):
+        super().__init__()
+        self.min_cardinality = min_cardinality
+        self.to_drop = []
+
+    @annotate("drop_low_cardinality", color="darkgreen", domain="nvt_python")
+    def transform(self, col_selector: ColumnSelector, df: DataFrameType) -> DataFrameType:
+        return df.drop(self.to_drop, axis=1)
+
+    def compute_output_schema(self, input_schema, selector, prev_output_schema=None):
+        output_columns = []
+        for col in input_schema:
+            if Tags.CATEGORICAL in col.tags:
+                domain = col.int_domain
+                if domain and domain.max <= self.min_cardinality:
+                    self.to_drop.append(col.name)
+                    continue
+            output_columns.append(col)
+        return Schema(output_columns)


Test patch:
diff --git a/tests/unit/ops/test_drop_low_cardinality.py b/tests/unit/ops/test_drop_low_cardinality.py
new file mode 100644
index 00000000000..275a9e3f26b
--- /dev/null
+++ b/tests/unit/ops/test_drop_low_cardinality.py
@@ -0,0 +1,50 @@
+#
+# Copyright (c) 2021, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import pandas as pd
+import pytest
+
+import nvtabular as nvt
+from tests.conftest import assert_eq
+
+try:
+    import cudf
+
+    _CPU = [True, False]
+except ImportError:
+    _CPU = [True]
+
+
+@pytest.mark.parametrize("cpu", _CPU)
+def test_drop_low_cardinality(tmpdir, cpu):
+    df = pd.DataFrame()
+    if not cpu:
+        df = cudf.DataFrame(df)
+
+    df["col1"] = ["a", "a", "a", "a", "a"]
+    df["col2"] = ["a", "a", "a", "a", "b"]
+    df["col3"] = ["a", "a", "b", "b", "c"]
+
+    features = list(df.columns) >> nvt.ops.Categorify() >> nvt.ops.DropLowCardinality()
+
+    workflow = nvt.Workflow(features)
+    transformed = workflow.fit_transform(nvt.Dataset(df)).to_ddf().compute()
+
+    assert workflow.output_schema.column_names == ["col2", "col3"]
+
+    expected = df.drop(["col1"], axis=1)
+    expected["col2"] = [1, 1, 1, 1, 2]
+    expected["col3"] = [1, 1, 2, 2, 3]
+    assert_eq(transformed, expected)