Repository: pyjanitor-devs/pyjanitor

Issue:
Forward/Backward Fill
# Brief Description

I would like to propose a forward/backward fill method, similar to ``fill`` in tidyr. In Pandas, forward/backward fill can be achieved, but you cannot select directions for different columns within a dataframe.

# Example API
    @pf.register_dataframe_method
    def fill(df, columns = ..., directions = sequence of directions(up, down, updown, downup):
        """
        Forward or backward fill on a column or columns of a dataframe.
        """
        df[column] = df[column].ffill() #forward fill will be the default
        return df
<!-- One of the selling points of pyjanitor is the API. Hence, we guard the API very carefully, and want to
make sure that it is accessible and understandable to many people. Please provide a few examples of what the API
of the new function you're proposing will look like. We have provided an example that you should modify. -->


```python
example data : 
df = pd.DataFrame(data={'a':[1,2,3,None],'b':[4,5,None,6],'c':[None,None,7,8]})

          a	b	c
0	1.0	4.0	NaN
1	2.0	5.0	NaN
2	3.0	NaN	7.0
3	NaN	6.0	8.0

#downward fill (backward fill in Pandas) on only the `a` column:
df.fill(columns="a", directions = "down")

# downward fill on  ``a``, upward fill on  ``c`` :

# the idea is the directions will be paired with the columns
# similar to what exists when sorting multiple columns in Pandas' sort_values 
# with booleans passed to the ``ascending`` argument
df.fill(columns=["a", "c"], directions = ["down", "up"])



# currently in Pandas, ffill and bfill are applied to all the columns that have null values, 
# you cannot select which columns to be forward or backward filled.
```

Still undecided on the name, as I do not feel it clearly conveys what it should do without reading the function description. 


Discussion:
@samukweku I like the idea! I’ve had to use filling before, and the lack of a dataframe-attached API made things just a _tad_ more complicated. 
Glad to know it could be useful to others as well. I would love to work on it, but still need opinions from others as well, especially with the name. Maybe someone has something better?
To me, the function name `fill` sounds about right. The argument options could possibly use additional eyes.

I can see where the "up" and "down" comes from - it is natural to look at a table and say "fill upwards" vs. "fill downwards". 

At the same time, pandas has the "forward fill" and "backward fill", which I think come from time series analysis. There, forward corresponds to "fill downwards", because it is usually assumed that time axis flows top to bottom. 

I guess it's a matter of choosing one set of semantic conventions, sticking with it, and documenting other possible conventions to educate users?

Now that I typed all of the above out, I think it's good to get a few more opinions. Others?

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -4104,3 +4104,144 @@ def process_text(
     df[column] = getattr(df[column].str, string_function)(*args, **kwargs)
 
     return df
+
+
+@pf.register_dataframe_method
+def fill_direction(
+    df: pd.DataFrame,
+    directions: Dict[Hashable, str],
+    limit: Optional[int] = None,
+) -> pd.DataFrame:
+    """Provide a method-chainable function for filling missing values
+    in selected columns.
+
+    Missing values are filled using the next or previous entry.
+    The columns are paired with the directions in a dictionary.
+    It is a wrapper for ``pd.Series.ffill`` and ``pd.Series.bfill``.
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = pd.DataFrame({"text": ["ragnar", np.nan, "sammywemmy",
+                                    np.nan, "ginger"],
+                           "code" : [np.nan, 2, 3, np.nan, 5]})
+
+        # Single column :
+        df.fill_direction({"text" : "up"})
+        # text       |   code
+        # ragnar     |    NaN
+        # sammywemmy |    2
+        # sammywemmy |    3
+        # ginger     |    NaN
+        # ginger     |    5
+
+        # Multiple columns :
+        df.fill_direction({"text" : "down", "code" : "down"})
+
+        # text       |   code
+        # ragnar     |    NaN
+        # ragnar     |    2
+        # sammywemmy |    3
+        # sammywemmy |    3
+        # ginger     |    5
+
+        # Multiple columns in different directions.
+        df.fill_direction({"text" : "up", "code" : "down"})
+
+        # text       |   code
+        # ragnar     |    NaN
+        # sammywemmy |    2
+        # sammywemmy |    3
+        # ginger     |    3
+        # ginger     |    5
+
+    Functional usage syntax:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = pd.DataFrame(...)
+        df = jn.fill_direction(
+            df = df,
+            directions = {column_1 : direction_1, column_2 : direction_2, ...},
+            limit = None # limit must be greater than 0
+            )
+
+    Method-chaining usage syntax:
+
+    .. code-block:: python
+
+        import pandas as pd
+        import janitor as jn
+
+        df = (
+            pd.DataFrame(...)
+            .fill_direction(
+            directions = {column_1 : direction_1, column_2 : direction_2, ...},
+            limit = None # limit must be greater than 0
+            )
+        )
+
+    :param df: A pandas dataframe.
+    :param directions: Key - value pairs of columns and directions. Directions
+        can be either `down`(default), `up`, `updown`(fill up then down) and
+        `downup` (fill down then up).
+    :param limit: number of consecutive null values to forward/backward fill.
+        Value must be greater than 0.
+    :returns: A pandas dataframe with modified column(s).
+    :raises: ValueError if ``directions`` dictionary is empty.
+    :raises: ValueError if column supplied is not in the dataframe.
+    :raises: ValueError if direction supplied is not one of `down`,`up`,
+        `updown`, or `downup`.
+    """
+
+    # check that dictionary is not empty
+    if not directions:
+        raise ValueError("A mapping of columns with directions is required.")
+
+    # check that the right columns are provided
+    # should be removed once the minimum Pandas version is 1.1,
+    # as Pandas loc will raise a KeyError if columns provided do not exist
+    wrong_columns_provided = set(directions).difference(df.columns)
+    if any(wrong_columns_provided):
+        if len(wrong_columns_provided) > 1:
+            outcome = ", ".join(f"'{word}'" for word in wrong_columns_provided)
+            raise ValueError(
+                f"Columns {outcome} do not exist in the dataframe."
+            )
+        outcome = "".join(wrong_columns_provided)
+        raise ValueError(f"Column {outcome} does not exist in the dataframe.")
+
+    # check that the right directions are provided
+    set_directions = {"up", "down", "updown", "downup"}
+
+    # linter throws an error when I use dictionary.values()
+    # it assumes that dictionary is a dataframe
+    directions_values = [value for key, value in directions.items()]
+    wrong_directions_provided = set(directions_values).difference(
+        set_directions
+    )
+    if any(wrong_directions_provided):
+        raise ValueError(
+            """The direction should be a string and should be one of `up`,
+            `down`, `updown`, or `downup`."""
+        )
+
+    for column, direction in directions.items():
+        if direction == "up":
+            df.loc[:, column] = df.loc[:, column].bfill(limit=limit)
+        elif direction == "down":
+            df.loc[:, column] = df.loc[:, column].ffill(limit=limit)
+        elif direction == "updown":
+            df.loc[:, column] = (
+                df.loc[:, column].bfill(limit=limit).ffill(limit=limit)
+            )
+        elif direction == "downup":
+            df.loc[:, column] = (
+                df.loc[:, column].ffill(limit=limit).bfill(limit=limit)
+            )
+    return df


Test patch:
diff --git a/tests/functions/test_fill_direction.py b/tests/functions/test_fill_direction.py
new file mode 100644
--- /dev/null
+++ b/tests/functions/test_fill_direction.py
@@ -0,0 +1,145 @@
+import numpy as np
+import pandas as pd
+import pytest
+from pandas.testing import assert_frame_equal
+
+
+@pytest.fixture
+def df():
+    return pd.DataFrame(
+        [
+            {
+                "rank": 1,
+                "pet_type": np.nan,
+                "breed": "Boston Terrier",
+                "owner": "sam",
+            },
+            {
+                "rank": 2,
+                "pet_type": np.nan,
+                "breed": "Retrievers (Labrador)",
+                "owner": "ogor",
+            },
+            {
+                "rank": 3,
+                "pet_type": np.nan,
+                "breed": "Retrievers (Golden)",
+                "owner": "nathan",
+            },
+            {
+                "rank": 4,
+                "pet_type": np.nan,
+                "breed": "French Bulldogs",
+                "owner": np.nan,
+            },
+            {
+                "rank": 5,
+                "pet_type": np.nan,
+                "breed": "Bulldogs",
+                "owner": np.nan,
+            },
+            {
+                "rank": 6,
+                "pet_type": "Dog",
+                "breed": "Beagles",
+                "owner": np.nan,
+            },
+            {
+                "rank": 1,
+                "pet_type": np.nan,
+                "breed": "Persian",
+                "owner": np.nan,
+            },
+            {
+                "rank": 2,
+                "pet_type": np.nan,
+                "breed": "Maine Coon",
+                "owner": "ragnar",
+            },
+            {
+                "rank": 3,
+                "pet_type": np.nan,
+                "breed": "Ragdoll",
+                "owner": np.nan,
+            },
+            {
+                "rank": 4,
+                "pet_type": np.nan,
+                "breed": "Exotic",
+                "owner": np.nan,
+            },
+            {
+                "rank": 5,
+                "pet_type": np.nan,
+                "breed": "Siamese",
+                "owner": np.nan,
+            },
+            {
+                "rank": 6,
+                "pet_type": "Cat",
+                "breed": "American Short",
+                "owner": "adaora",
+            },
+        ]
+    )
+
+
+def test_fill_column(df):
+    """Fill down on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].ffill()
+    assert_frame_equal(df.fill_direction({"pet_type": "down"}), expected)
+
+
+def test_fill_column_up(df):
+    """Fill up on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].bfill()
+    assert_frame_equal(df.fill_direction({"pet_type": "up"}), expected)
+
+
+def test_fill_column_updown(df):
+    """Fill upwards, then downwards on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].bfill().ffill()
+    assert_frame_equal(df.fill_direction({"pet_type": "updown"}), expected)
+
+
+def test_fill_column_down_up(df):
+    """Fill downwards, then upwards on a single column."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].ffill().bfill()
+    assert_frame_equal(df.fill_direction({"pet_type": "downup"}), expected)
+
+
+def test_fill_multiple_columns(df):
+    """Fill on multiple columns with a single direction."""
+    expected = df.copy()
+    expected.loc[:, ["pet_type", "owner"]] = expected.loc[
+        :, ["pet_type", "owner"]
+    ].ffill()
+    assert_frame_equal(
+        df.fill_direction({"pet_type": "down", "owner": "down"}), expected
+    )
+
+
+def test_fill_multiple_columns_multiple_directions(df):
+    """Fill on multiple columns with different directions."""
+    expected = df.copy()
+    expected.loc[:, "pet_type"] = expected.loc[:, "pet_type"].ffill()
+    expected.loc[:, "owner"] = expected.loc[:, "owner"].bfill()
+    assert_frame_equal(
+        df.fill_direction({"pet_type": "down", "owner": "up"}), expected
+    )
+
+
+def test_wrong_column_name(df):
+    """Raise Value Error if wrong column name is provided."""
+    with pytest.raises(ValueError):
+        df.fill_direction({"PetType": "down"})
+
+
+def test_wrong_direction(df):
+    """Raise Value Error if wrong direction is provided."""
+    with pytest.raises(ValueError):
+        df.fill_direction({"pet_type": "upanddawn"})
diff --git a/tests/functions/test_filter_date.py b/tests/functions/test_filter_date.py
--- a/tests/functions/test_filter_date.py
+++ b/tests/functions/test_filter_date.py
@@ -3,6 +3,9 @@
 import pytest
 
 
+@pytest.mark.xfail
+# pandas 1.1 raises a KeyError if columns/indices passed to loc does not exist
+# pandas <1.1 raises a TypeError
 def test_filter_date_column_name(date_dataframe):
     df = date_dataframe
     # `column_name` wasn't a string