Repository: pyjanitor-devs/pyjanitor

Issue:
[ENH] Add ability to flag null values in columns
# Brief Description

Hello everyone, I'd like to suggest a new method (I'll tentatively call it `flag_nulls`) that adds a new column to the dataframe to indicate if there are null values in the row.

If you are preparing a dataframe for machine learning or something, it's important to fill in the null values with something. However, the fact that the data is null could, in fact, be its own feature--for example, if someone is submitting an insurance claim and they haven't provided important information about themselves, that might be a flag that the claim is fraudulent.

# Example API

```python
# flag null values in columns
df.flag_nulls(column_name='null_flag', columns=None)
# column_name gives the name of the new column we generate
# columns is a list of the columns that we check for null values. If columns is None, we use all the columns

df1 = pd.DataFrame({'a': [1, 2, None, 4], 'b': [5.0, None, 7.0, 8.0]}

print df1.flag_nulls()
```
a | b | null_flag
---|---|---
1    | 5.0 | 0
2   | None | 1
None | 7.0 | 1
4 | 8.0 | 0

```python
print df1.flag_nulls(columns=['a'])
```
a | b | null_flag
---|---|---
1    | 5.0 | 0
2   | None | 0
None | 7.0 | 1
4 | 8.0 | 0

```python
print df1.flag_nulls(columns=['a'], column_name='flag')
```
a | b | flag
---|---|---
1    | 5.0 | 0
2   | None | 0
None | 7.0 | 1
4 | 8.0 | 0

# Notes

- Computerphile [agrees with me](https://www.youtube.com/watch?v=oCQbC818KKU) on this strategy as well ;)
- I would like to work on this issue if that's acceptable.


Discussion:
Seems like a useful feature to implement, especially if it could be vectorized.

When I do something like this, I do a slow `apply` method.
```python
df1['null_flag'] = df1.apply(lambda row: any(row.isna()), axis=1).astype(int)
```
Thanks @anzelpwj! I have marked this as being worked on, looking forward to seeing your PR!
Okay, I tested several different approaches.  I created some dataframes of random values, and set 5% of entries to be null. First testing was on a dataframe of 10 columns and 10k rows.

Using the apply function with
```python
df.apply(lambda row: any(row[list_of_cols].isna()), axis=1)
```
and
```python
df.apply(lambda row: pd.isnull(row[list_of_cols]).any(), axis=1)
```
Were about equivalent, and very slow.

Two much faster approaches are as follows:
```python
# Option 1
null_array = np.zeros(len(df))
for col in list_of_cols:
    null_array = np.logical_or(
        null_array,
        pd.isnull(df[col]))
# Option 2
pd.isnull(df[list_of_cols]).any(axis=1)
```
Using `%timeit` on my laptop reveals the following:

| num_rows | num_cols | time numpy logical_or | time pd isnull any |
|----------|--------|-----|----|
|10000 | 10 | 2.2 ms | 2.36 ms |
|30000 | 10 | 2.6 ms | 5.13 ms |
|100000 | 10 | 3.74 ms | 15.3 ms |
|300000 | 10 | 6 ms | 43.6 ms |
|10000 | 100 | 21.6 ms | 12.6 ms |
|10000 | 1000 | 220 ms | 126 ms |
|30000 | 100 | 24.4 ms | 37.2 ms |

Given that dataframes are probably going to be much longer than they are wide (and people may be using this on just a few columns), I'm inclined to say that using `logical_or` is probably the way to go. If we want to get really tricky, I could probably do something to compare the width to the length, but I'm guessing that might be overthinking things for now. I'll go ahead with just the `logical_or`.
Hmmm, I just thought to try `np.any(pd.isnull(df[list_of_cols]), axis=1)` instead of the pandas any, and this seems to be somewhat faster than the pandas approach, but still fails compared to the numpy logical_or for long, thin tables. Worth keeping in mind though...

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -3309,3 +3309,76 @@ def take_a_if_even(x):
     """
     df = df.copy().join(df.apply(func, axis=1).rename(new_column_name))
     return df
+
+
+@pf.register_dataframe_method
+def flag_nulls(
+    df: pd.DataFrame,
+    column_name: str = "null_flag",
+    columns: Union[str, Iterable[str], Any] = None,
+) -> pd.DataFrame:
+    """
+    Creates a new column to indicate whether you have null values in a given
+    row. If the columns parameter is not set, looks across the entire
+    DataFrame, otherwise will look only in the columns you set.
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        data = pd.DataFrame(
+            {'a': [1, 2, None, 4],
+             'b': [5.0, None, 7.0, 8.0]})
+
+        df.flag_nulls()
+        #  'a' | 'b'  | 'null_flag'
+        #   1  | 5.0  |   0
+        #   2  | None |   1
+        # None | 7.0  |   1
+        #   4  | 8.0  |   0
+
+        jn.functions.flag_nulls(data)
+        #  'a' | 'b'  | 'null_flag'
+        #   1  | 5.0  |   0
+        #   2  | None |   1
+        # None | 7.0  |   1
+        #   4  | 8.0  |   0
+
+        df.flag_nulls(columns=['b'])
+        #  'a' | 'b'  | 'null_flag'
+        #   1  | 5.0  |   0
+        #   2  | None |   1
+        # None | 7.0  |   0
+        #   4  | 8.0  |   0
+
+
+    :param df: Input Pandas dataframe.
+    :param column_name: Name for the output column. Defaults to 'null_flag'.
+    :param columns: List of columns to look at for finding null values. If you
+        only want to look at one column, you can simply give its name. If set
+        to None (default), all DataFrame columns are used.
+    :returns: Input dataframe with the null flag column.
+    :raises: ValueError
+    """
+    # Sort out columns input
+    if isinstance(columns, str):
+        columns = [columns]
+    elif columns is None:
+        columns = df.columns
+    elif not isinstance(columns, Iterable):
+        # Handle cases where we have an integer column or something
+        columns = [columns]
+
+    # Input sanitation checks
+    check_column(df, columns)
+    check_column(df, [column_name], present=False)
+
+    # This algorithm works best for n_rows >> n_cols. See issue #501
+    null_array = np.zeros(len(df))
+    for col in columns:
+        null_array = np.logical_or(null_array, pd.isnull(df[col]))
+
+    df = df.copy()
+    df[column_name] = null_array.astype(int)
+    return df
diff --git a/janitor/utils.py b/janitor/utils.py
--- a/janitor/utils.py
+++ b/janitor/utils.py
@@ -249,9 +249,32 @@ def rename_kwargs(func_name: str, kwargs: Dict, aliases: Dict):
             kwargs[new_alias] = kwargs.pop(old_alias)
 
 
-def check_column(df: pd.DataFrame, old_column_names: List):
+def check_column(
+    df: pd.DataFrame, old_column_names: List, present: bool = True
+):
+    """
+    One-liner syntactic sugar for checking the presence or absence of a column.
+
+    Should be used like this::
+
+        check(df, ['a', 'b'], present=True)
+
+    :param df: The name of the variable.
+    :param old_column_names: A list of column names we want to check to see if
+        present (or absent) in df.
+    :param present: If True (default), checks to see if all of old_column_names
+        are in df.columns. If False, checks that none of old_column_names are
+        in df.columns.
+    :returns: ValueError if data is not the expected type.
+    """
     for column_name in old_column_names:
-        if column_name not in df.columns:
-            raise ValueError(
-                f"{column_name} not present in dataframe columns!"
-            )
+        if present:
+            if column_name not in df.columns:
+                raise ValueError(
+                    f"{column_name} not present in dataframe columns!"
+                )
+        else:  # Tests for exclusion
+            if column_name in df.columns:
+                raise ValueError(
+                    f"{column_name} already present in dataframe columns!"
+                )


Test patch:
diff --git a/tests/functions/test_flag_nulls.py b/tests/functions/test_flag_nulls.py
new file mode 100644
--- /dev/null
+++ b/tests/functions/test_flag_nulls.py
@@ -0,0 +1,66 @@
+import pytest
+from pandas.testing import assert_frame_equal
+
+from janitor.functions import flag_nulls
+
+
+@pytest.mark.functions
+def test_functional_on_all_columns(missingdata_df):
+    expected = missingdata_df.copy()
+    expected["null_flag"] = [0, 1, 1] * 3
+
+    df = missingdata_df.flag_nulls()
+
+    assert_frame_equal(df, expected)
+
+    # Should also be the same for explicit columns
+    df = missingdata_df.flag_nulls(columns=["a", "Bell__Chart"])
+
+    assert_frame_equal(df, expected)
+
+
+@pytest.mark.functions
+def test_non_method_functional(missingdata_df):
+    expected = missingdata_df.copy()
+    expected["null_flag"] = [0, 1, 1] * 3
+
+    df = flag_nulls(missingdata_df)
+
+    assert_frame_equal(df, expected)
+
+
+@pytest.mark.functions
+def test_functional_on_some_columns(missingdata_df):
+    expected = missingdata_df.copy()
+    expected["null_flag"] = [0, 0, 1] * 3
+
+    df = missingdata_df.flag_nulls(columns=["a"])
+
+    assert_frame_equal(df, expected)
+
+    # Testing when we provide the direct name
+    df = missingdata_df.flag_nulls(columns="a")
+
+    assert_frame_equal(df, expected)
+
+
+@pytest.mark.functions
+def test_rename_output_column(missingdata_df):
+    expected = missingdata_df.copy()
+    expected["flag"] = [0, 1, 1] * 3
+
+    df = missingdata_df.flag_nulls(column_name="flag")
+
+    assert_frame_equal(df, expected)
+
+
+@pytest.mark.functions
+def test_fail_column_name_in_columns(missingdata_df):
+    with pytest.raises(ValueError):
+        missingdata_df.flag_nulls(column_name="a")
+
+
+@pytest.mark.functions
+def test_fail_column_val_not_in_columns(missingdata_df):
+    with pytest.raises(ValueError):
+        missingdata_df.flag_nulls(columns=["c"])
diff --git a/tests/utils/test_check_column.py b/tests/utils/test_check_column.py
--- a/tests/utils/test_check_column.py
+++ b/tests/utils/test_check_column.py
@@ -1,5 +1,4 @@
 import pytest
-from hypothesis import given
 
 from janitor.utils import check_column
 
@@ -7,7 +6,7 @@
 @pytest.mark.utils
 def test_check_column(dataframe):
     """
-    rename_column should return true if column exist
+    check_column should return if column exist
     """
     assert check_column(dataframe, ["a"]) is None
 
@@ -15,7 +14,25 @@ def test_check_column(dataframe):
 @pytest.mark.utils
 def test_check_column_absent_column(dataframe):
     """
-    rename_column should raise an error if the column is absent.
+    check_column should raise an error if the column is absent.
     """
     with pytest.raises(ValueError):
         check_column(dataframe, ["b"])
+
+
+@pytest.mark.utils
+def test_check_column_excludes(dataframe):
+    """
+    check_column should return if column is absent and present is False
+    """
+    assert check_column(dataframe, ["b"], present=False) is None
+
+
+@pytest.mark.utils
+def test_check_column_absent_column_excludes(dataframe):
+    """
+    check_column should raise an error if the column is absent and present is
+    False
+    """
+    with pytest.raises(ValueError):
+        check_column(dataframe, ["a"], present=False)