Repository: ericmjl/pyjanitor (python)

Issue:
Forward/Backward Fill
# Brief Description

I would like to propose a forward/backward fill method, similar to ``fill`` in tidyr. In Pandas, forward/backward fill can be achieved, but you cannot select directions for different columns within a dataframe.

# Example API
    @pf.register_dataframe_method
    def fill(df, columns = ..., directions = sequence of directions(up, down, updown, downup):
        """
        Forward or backward fill on a column or columns of a dataframe.
        """
        df[column] = df[column].ffill() #forward fill will be the default
        return df
<!-- One of the selling points of pyjanitor is the API. Hence, we guard the API very carefully, and want to
make sure that it is accessible and understandable to many people. Please provide a few examples of what the API
of the new function you're proposing will look like. We have provided an example that you should modify. -->


```python
example data : 
df = pd.DataFrame(data={'a':[1,2,3,None],'b':[4,5,None,6],'c':[None,None,7,8]})

          a	b	c
0	1.0	4.0	NaN
1	2.0	5.0	NaN
2	3.0	NaN	7.0
3	NaN	6.0	8.0

#downward fill (backward fill in Pandas) on only the `a` column:
df.fill(columns="a", directions = "down")

# downward fill on  ``a``, upward fill on  ``c`` :

# the idea is the directions will be paired with the columns
# similar to what exists when sorting multiple columns in Pandas' sort_values 
# with booleans passed to the ``ascending`` argument
df.fill(columns=["a", "c"], directions = ["down", "up"])



# currently in Pandas, ffill and bfill are applied to all the columns that have null values, 
# you cannot select which columns to be forward or backward filled.
```

Still undecided on the name, as I do not feel it clearly conveys what it should do without reading the function description. 

Pull request:
[ENH] Pyjanitor fill method
# PR Description

Please describe the changes proposed in the pull request:

- A  method-chainable ``fill`` method  for forward and backward fills on selected columns of a dataframe.

**This PR resolves #700 .**

Please ensure that you have done the following:

1. [x] PR in from a fork off your branch. Do not PR from `<your_username>`:`dev`, but rather from `<your_username>`:`<feature-branch_name>`.
<!-- Doing this helps us keep the commit history much cleaner than it would otherwise be. -->
2. [x] If you're not on the contributors list, add yourself to `AUTHORS.rst`.
<!-- We'd like to acknowledge your contributions! -->
3. [ ] Add a line to `CHANGELOG.rst` under the latest version header (i.e. the one that is "on deck") describing the contribution.
    - Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

## Quick Check

To do a very quick check that everything is correct, follow these steps below:

- [x] Run the command `make check` from pyjanitor's top-level directory. This will automatically run:
    - black formatting
    - flake8 checking
    - running the test suite
    - docs build

Once done, please check off the check-box above.

If `make check` does not work for you, you can execute the commands listed in the Makefile individually.

## Code Changes

<!-- If you have not made code changes, please feel free to delete this section. -->

If you are adding code changes, please ensure the following:

- [x] Ensure that you have added tests.
- [x] Run all tests (`$ pytest .`) locally on your machine.
    - [x] Check to ensure that test coverage covers the lines of code that you have added.
    - [x] Ensure that all tests pass.

## Documentation Changes

<!-- If you have not made documentation changes, please feel free to delete this section. -->

If you are adding documentation changes, please ensure the following:

- [ ] Build the docs locally.
- [ ] View the docs to check that it renders correctly.

# Relevant Reviewers

<!-- Finally, please tag relevant maintainers to review. -->

Please tag maintainers to review.

- @ericmjl


Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 3ebfed2..fe444dc 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -14,6 +14,8 @@ new version (on deck)
 - [ENH] Added ``process_text`` wrapper function for all Pandas string methods. @samukweku
 - [TST] Only skip tests for non-installed libraries on local machine. @hectormz
 - [DOC] Fix minor issues in documentation. @hectormz
+- [ENH] Added ``fill_direction`` function for forward/backward fills on missing values
+  for selected columns in a dataframe. @samukweku
 
 
 v0.20.7
diff --git a/docs/reference/general_functions.rst b/docs/reference/general_functions.rst
index 736ee0c..0f19332 100644
--- a/docs/reference/general_functions.rst
+++ b/docs/reference/general_functions.rst
@@ -28,6 +28,7 @@ Modify columns
     groupby_agg
     join_apply
     drop_duplicate_columns
+    process_text
 
 Modify values
 ~~~~~~~~~~~~~
@@ -35,6 +36,7 @@ Modify values
     :toctree: janitor.functions/
 
     fill_empty
+    fill_direction
     convert_excel_date
     convert_matlab_date
     convert_unix_date
@@ -86,5 +88,4 @@ Other
     move
     toset
     unionize_dataframe_categories
-    process_text
     
diff --git a/janitor/functions.py b/janitor/functions.py
index d93d60b..fe85a89 100644
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -4104,3 +4104,144 @@ def process_text(
     df[column] = getattr(df[column].str, string_function)(*args, **kwargs)
 
     return df
+
+
+@pf.register_dataframe_method
+def fill_direction(
+    df: pd.DataFrame,
+    directions: Dict[Hashable, str],
+    limit: Optional[int] = None,
+) -> pd.DataFrame:
+    """Provide a method-chainable function for filling missing values
+    in selected columns.
+
+    Missing values are filled using the next or previous entry.
+    The columns are paired with the directions in a dictionary.
+    It is a wrapper for ``pd.Series.ffill`` and ``pd.Series.bfill``.
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = pd.DataFrame({"text": ["ragnar", np.nan, "sammywemmy",
+                                    np.nan, "ginger"],
+                           "code" : [np.nan, 2, 3, np.nan, 5]})
+
+        # Single column :
+        df.fill_direction({"text" : "up"})
+        # text       |   code
+        # ragnar     |    NaN
+        # sammywemmy |    2
+        # sammywemmy |    3
+        # ginger     |    NaN
+        # ginger     |    5
+
+        # Multiple columns :
+        df.fill_direction({"text" : "down", "code" : "down"})
+
+        # text       |   code
+        # ragnar     |    NaN
+        # ragnar     |    2
+        # sammywemmy |    3
+        # sammywemmy |    3
+        # ginger     |    5
+
+        # Multiple columns in different directions.
+        df.fill_direction({"text" : "up", "code" : "down"})
+
+        # text       |   code
+        # ragnar     |    NaN
+        # sammywemmy |    2
+        # sammywemmy |    3
+        # ginger     |    3
+        # ginger     |    5
+
+    Functional usage syntax:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = pd.DataFrame(...)
+        df = jn.fill_direction(
+            df = df,
+            directions = {column_1 : direction_1, column_2 : direction_2, ...},
+            limit = None # limit must be greater than 0
+            )
+
+    Method-chaining usage syntax:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = (
+            pd.DataFrame(...)
+            .fill_direction(
+            directions = {column_1 : direction_1, column_2 : direction_2, ...},
+            limit = None # limit must be greater than 0
+            )
+        )
+
+    :param df: A pandas dataframe.
+    :param directions: Key - value pairs of columns and directions. Directions
+        can be either `down`(default), `up`, `updown`(fill up then down) and
+        `downup` (fill down then up).
+    :param limit: number of consecutive null values to forward/backward fill.
+        Value must be greater than 0.
+    :returns: A pandas dataframe with modified column(s).
+    :raises: ValueError if ``directions`` dictionary is empty.
+    :raises: ValueError if column supplied is not in the dataframe.
+    :raises: ValueError if direction supplied is not one of `down`,`up`,
+        `updown`, or `downup`.
+    """
+
+    # check that dictionary is not empty
+    if not directions:
+        raise ValueError("A mapping of columns with directions is required.")
+
+    # check that the right columns are provided
+    # should be removed once the minimum Pandas version is 1.1,
+    # as Pandas loc will raise a KeyError if columns provided do not exist
+    wrong_columns_provided = set(directions).difference(df.columns)
+    if any(wrong_columns_provided):
+        if len(wrong_columns_provided) > 1:
+            outcome = ", ".join(f"'{word}'" for word in wrong_columns_provided)
+            raise ValueError(
+                f"Columns {outcome} do not exist in the dataframe."
+            )
+        outcome = "".join(wrong_columns_provided)
+        raise ValueError(f"Column {outcome} does not exist in the dataframe.")
+
+    # check that the right directions are provided
+    set_directions = {"up", "down", "updown", "downup"}
+
+    # linter throws an error when I use dictionary.values()
+    # it assumes that dictionary is a dataframe
+    directions_values = [value for key, value in directions.items()]
+    wrong_directions_provided = set(directions_values).difference(
+        set_directions
+    )
+    if any(wrong_directions_provided):
+        raise ValueError(
+            """The direction should be a string and should be one of `up`,
+            `down`, `updown`, or `downup`."""
+        )
+
+    for column, direction in directions.items():
+        if direction == "up":
+            df.loc[:, column] = df.loc[:, column].bfill(limit=limit)
+        elif direction == "down":
+            df.loc[:, column] = df.loc[:, column].ffill(limit=limit)
+        elif direction == "updown":
+            df.loc[:, column] = (
+                df.loc[:, column].bfill(limit=limit).ffill(limit=limit)
+            )
+        elif direction == "downup":
+            df.loc[:, column] = (
+                df.loc[:, column].ffill(limit=limit).bfill(limit=limit)
+            )
+    return df


Test patch:
diff --git a/tests/functions/test_fill_direction.py b/tests/functions/test_fill_direction.py
new file mode 100644
index 0000000..20a6056
--- /dev/null
+++ b/tests/functions/test_fill_direction.py
@@ -0,0 +1,145 @@
+import numpy as np
+import pandas as pd
+import pytest
+from pandas.testing import assert_frame_equal
+
+
+@pytest.fixture
+def df():
+    return pd.DataFrame(
+        [
+            {
+                "rank": 1,
+                "pet_type": np.nan,
+                "breed": "Boston Terrier",
+                "owner": "sam",
+            },
+            {
+                "rank": 2,
+                "pet_type": np.nan,
+                "breed": "Retrievers (Labrador)",
+                "owner": "ogor",
+            },
+            {
+                "rank": 3,
+                "pet_type": np.nan,
+                "breed": "Retrievers (Golden)",
+                "owner": "nathan",
+            },
+            {
+                "rank": 4,
+                "pet_type": np.nan,
+                "breed": "French Bulldogs",
+                "owner": np.nan,
+            },
+            {
+                "rank": 5,
+                "pet_type": np.nan,
+                "breed": "Bulldogs",
+                "owner": np.nan,
+            },
+            {
+                "rank": 6,
+                "pet_type": "Dog",
+                "breed": "Beagles",
+                "owner": np.nan,
+            },
+            {
+                "rank": 1,
+                "pet_type": np.nan,
+                "breed": "Persian",
+                "owner": np.nan,
+            },
+            {
+                "rank": 2,
+                "pet_type": np.nan,
+                "breed": "Maine Coon",
+                "owner": "ragnar",
+            },
+            {
+                "rank": 3,
+                "pet_type": np.nan,
+                "breed": "Ragdoll",
+                "owner": np.nan,
+            },
+            {
+                "rank": 4,
+                "pet_type": np.nan,
+                "breed": "Exotic",
+                "owner": np.nan,
+            },
+            {
+                "rank": 5,
+                "pet_type": np.nan,
+                "breed": "Siamese",
+                "owner": np.nan,
+            },
+            {
+                "rank": 6,
+                "pet_type": "Cat",
+                "breed": "American Short",
+                "owner": "adaora",
+            },
+        ]
+    )
+
+
+def test_fill_column(df):
+    """Fill down on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].ffill()
+    assert_frame_equal(df.fill_direction({"pet_type": "down"}), expected)
+
+
+def test_fill_column_up(df):
+    """Fill up on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].bfill()
+    assert_frame_equal(df.fill_direction({"pet_type": "up"}), expected)
+
+
+def test_fill_column_updown(df):
+    """Fill upwards, then downwards on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].bfill().ffill()
+    assert_frame_equal(df.fill_direction({"pet_type": "updown"}), expected)
+
+
+def test_fill_column_down_up(df):
+    """Fill downwards, then upwards on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].ffill().bfill()
+    assert_frame_equal(df.fill_direction({"pet_type": "downup"}), expected)
+
+
+def test_fill_multiple_columns(df):
+    """Fill on multiple columns with a single direction."""
+    expected = df.copy()
+    expected.loc[:, ["pet_type", "owner"]] = expected.loc[
+        :, ["pet_type", "owner"]
+    ].ffill()
+    assert_frame_equal(
+        df.fill_direction({"pet_type": "down", "owner": "down"}), expected
+    )
+
+
+def test_fill_multiple_columns_multiple_directions(df):
+    """Fill on multiple columns with different directions."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].ffill()
+    expected.loc[:, "owner"] = expected.loc[:, "owner"].bfill()
+    assert_frame_equal(
+        df.fill_direction({"pet_type": "down", "owner": "up"}), expected
+    )
+
+
+def test_wrong_column_name(df):
+    """Raise Value Error if wrong column name is provided."""
+    with pytest.raises(ValueError):
+        df.fill_direction({"PetType": "down"})
+
+
+def test_wrong_direction(df):
+    """Raise Value Error if wrong direction is provided."""
+    with pytest.raises(ValueError):
+        df.fill_direction({"pet_type": "upanddawn"})
diff --git a/tests/functions/test_filter_date.py b/tests/functions/test_filter_date.py
index 64342da..a66ec14 100644
--- a/tests/functions/test_filter_date.py
+++ b/tests/functions/test_filter_date.py
@@ -3,6 +3,9 @@ import pandas as pd
 import pytest
 
 
+@pytest.mark.xfail
+# pandas 1.1 raises a KeyError if columns/indices passed to loc does not exist
+# pandas <1.1 raises a TypeError
 def test_filter_date_column_name(date_dataframe):
     df = date_dataframe
     # `column_name` wasn't a string