Repository: pyjanitor-devs/pyjanitor

Issue:
Add the ability to flag and fill if timestamps are missing in a pandas data frame
# Brief Description

Following up on https://github.com/ericmjl/pyjanitor/issues/703, this issue seeks to introduce the ability to flag if timestamps are missing in a given data frame, provided the frequency of the time series data in the data frame  

I would like to propose...

# Example API
```python
import pandas as pd 
import janitor 

def _flag_missing_timestamps(df, frequency, first_time_stamp=None, last_time_stamp=None):
    """
    Utility function to test that if input time_series is missing any timestamps relative to expected timestamps
    generated based on the first_time_stamp, last_time_stamp and frequency.

    :param pd.DataFrame df: data frame which needs to be tested for missing timestamps
    :param int frequency: frequency i.e. sampling frequency of the data, expressed in seconds. For e.g. if the data has
    hourly frequency, then a value of 60 * 60 = 3600 is expected
    :param pd.Timestamp first_time_stamp: timestamp at which the time_series is expected to start from. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :param pd.Timestamp last_time_stamp: timestamp at which the time_series is expected to end with. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :return: boolean flag indicating whether there are any missing timestamps in the input time_series
    :rtype: bool
    """

@pf.register_dataframe_method
def fill_missing_timestamps(df, frequency, column_name=None, first_time_stamp=None, last_time_stamp=None): 
    """
    Re-index the data frame if there are missing timestamps found in the data frame.  
    
    :param pd.DataFrame df: 
    :param int frequency: frequency i.e. sampling frequency of the data, expressed in seconds. For e.g. if the data has
    hourly frequency, then a value of 60 * 60 = 3600 is expected
    :param str column_name: name of the column which has time series if not the index. Defaults to None. By default, the 
    index is used for checking for the timestamps, unless a value other than None is assigned to column_name 
    :param pd.Timestamp first_time_stamp: timestamp at which the time_series is expected to start from. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :param pd.Timestamp last_time_stamp: timestamp at which the time_series is expected to end with. Defaults to None
    If no input is provided, assumes the minimum value in time_series 
    :return: reindexed data frame if it turns out to have missing timestamps, else returns the input data frame as is 
    :rtype: pd.DataFrame
    """

```

Add the ability to flag and fill if timestamps are missing in a pandas data frame
# Brief Description

Following up on https://github.com/ericmjl/pyjanitor/issues/703, this issue seeks to introduce the ability to flag if timestamps are missing in a given data frame, provided the frequency of the time series data in the data frame  

I would like to propose...

# Example API
```python
import pandas as pd 
import janitor 

def _flag_missing_timestamps(df, frequency, first_time_stamp=None, last_time_stamp=None):
    """
    Utility function to test that if input time_series is missing any timestamps relative to expected timestamps
    generated based on the first_time_stamp, last_time_stamp and frequency.

    :param pd.DataFrame df: data frame which needs to be tested for missing timestamps
    :param int frequency: frequency i.e. sampling frequency of the data, expressed in seconds. For e.g. if the data has
    hourly frequency, then a value of 60 * 60 = 3600 is expected
    :param pd.Timestamp first_time_stamp: timestamp at which the time_series is expected to start from. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :param pd.Timestamp last_time_stamp: timestamp at which the time_series is expected to end with. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :return: boolean flag indicating whether there are any missing timestamps in the input time_series
    :rtype: bool
    """

@pf.register_dataframe_method
def fill_missing_timestamps(df, frequency, column_name=None, first_time_stamp=None, last_time_stamp=None): 
    """
    Re-index the data frame if there are missing timestamps found in the data frame.  
    
    :param pd.DataFrame df: 
    :param int frequency: frequency i.e. sampling frequency of the data, expressed in seconds. For e.g. if the data has
    hourly frequency, then a value of 60 * 60 = 3600 is expected
    :param str column_name: name of the column which has time series if not the index. Defaults to None. By default, the 
    index is used for checking for the timestamps, unless a value other than None is assigned to column_name 
    :param pd.Timestamp first_time_stamp: timestamp at which the time_series is expected to start from. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :param pd.Timestamp last_time_stamp: timestamp at which the time_series is expected to end with. Defaults to None
    If no input is provided, assumes the minimum value in time_series 
    :return: reindexed data frame if it turns out to have missing timestamps, else returns the input data frame as is 
    :rtype: pd.DataFrame
    """

```



Discussion:
@ericmjl, 
        A question - do you propose I add a new module called "timeseries.py" or did you want the functions to be housed somewhere else? Until I hear back I will just start off with "timeseries.py" and refactor based on your guidance
@ericmjl, 
        A question - do you propose I add a new module called "timeseries.py" or did you want the functions to be housed somewhere else? Until I hear back I will just start off with "timeseries.py" and refactor based on your guidance

Resolving patch:
diff --git a/janitor/timeseries.py b/janitor/timeseries.py
new file mode 100644
--- /dev/null
+++ b/janitor/timeseries.py
@@ -0,0 +1,89 @@
+"""
+Time series-specific data testing and cleaning functions.
+"""
+
+import pandas as pd
+import pandas_flavor as pf
+from janitor import check
+
+
+@pf.register_dataframe_method
+def fill_missing_timestamps(
+    df: pd.DataFrame,
+    frequency: str,
+    first_time_stamp: pd.Timestamp = None,
+    last_time_stamp: pd.Timestamp = None,
+) -> pd.DataFrame:
+    """
+    Fill dataframe with missing timestamps based on a defined frequency.
+
+    If timestamps are missing,
+    this function will reindex the dataframe.
+    If timestamps are not missing,
+    then the function will return the dataframe unmodified.
+    Example usage:
+    .. code-block:: python
+
+        df = (
+            pd.DataFrame(...)
+            .fill_missing_timestamps(frequency="1H")
+        )
+
+    :param df: Dataframe which needs to be tested for missing timestamps
+    :param frequency: frequency i.e. sampling frequency of the data.
+        Acceptable frequency strings are available
+        `here <https://pandas.pydata.org/pandas-docs/stable/>`_
+        Check offset aliases under time series in user guide
+    :param first_time_stamp: timestamp expected to start from
+        Defaults to None.
+        If no input is provided assumes the minimum value in time_series
+    :param last_time_stamp: timestamp expected to end with.
+        Defaults to None.
+        If no input is provided, assumes the maximum value in time_series
+    :returns: dataframe that has a complete set of contiguous datetimes.
+    """
+    # Check all the inputs are the correct data type
+    check("frequency", frequency, [str])
+    check("first_time_stamp", first_time_stamp, [pd.Timestamp, type(None)])
+    check("last_time_stamp", last_time_stamp, [pd.Timestamp, type(None)])
+
+    if first_time_stamp is None:
+        first_time_stamp = df.index.min()
+    if last_time_stamp is None:
+        last_time_stamp = df.index.max()
+
+    # Generate expected timestamps
+    expected_timestamps = pd.date_range(
+        start=first_time_stamp, end=last_time_stamp, freq=frequency
+    )
+
+    return df.reindex(expected_timestamps)
+
+
+def _get_missing_timestamps(
+    df: pd.DataFrame,
+    frequency: str,
+    first_time_stamp: pd.Timestamp = None,
+    last_time_stamp: pd.Timestamp = None,
+) -> pd.DataFrame:
+    """
+    Return the timestamps that are missing in a dataframe.
+
+    This function takes in a dataframe,
+    and checks its index against a dataframe
+    that contains the expected timestamps.
+    Here, we assume that the expected timestamps
+    are going to be of a larger size
+    than the timestamps available in the input dataframe ``df``.
+
+    If there are any missing timestamps in the input dataframe,
+    this function will return those missing timestamps
+    from the expected dataframe.
+    """
+    expected_df = df.fill_missing_timestamps(
+        frequency, first_time_stamp, last_time_stamp
+    )
+
+    missing_timestamps = expected_df.index.difference(df.index)
+
+    return expected_df.loc[missing_timestamps]


Test patch:
diff --git a/tests/timeseries/test_fill_missing_timestamps.py b/tests/timeseries/test_fill_missing_timestamps.py
new file mode 100644
--- /dev/null
+++ b/tests/timeseries/test_fill_missing_timestamps.py
@@ -0,0 +1,48 @@
+import pytest
+import pandas as pd
+from random import randint
+from janitor.timeseries import fill_missing_timestamps, _get_missing_timestamps
+
+
+# Random data for testing
+@pytest.fixture
+def timeseries_dataframe() -> pd.DataFrame:
+    """
+    Returns a time series dataframe
+    """
+    ts_index = pd.date_range("1/1/2019", periods=1000, freq="1H")
+    v1 = [randint(1, 2000) for i in range(1000)]
+    test_df = pd.DataFrame({"v1": v1}, index=ts_index)
+    return test_df
+
+
+@pytest.mark.timeseries
+def test_fill_missing_timestamps(timeseries_dataframe):
+    """Test that filling missing timestamps works as expected."""
+    # Remove random row from the data frame
+    random_number = randint(1, len(timeseries_dataframe))
+    df1 = timeseries_dataframe.drop(timeseries_dataframe.index[random_number])
+
+    # Fill missing timestamps
+    result = fill_missing_timestamps(df1, frequency="1H")
+
+    # Testing if the missing timestamp has been filled
+    assert len(result) == len(timeseries_dataframe)
+
+    # Testing if indices are exactly the same after filling
+    original_index = timeseries_dataframe.index
+    new_index = result.index
+    delta = original_index.difference(new_index)
+
+    assert delta.empty is True
+
+
+@pytest.mark.timeseries
+def test__get_missing_timestamps(timeseries_dataframe):
+    """Test utility function for identifying the missing timestamps."""
+    from random import sample
+
+    timestamps_to_drop = sample(timeseries_dataframe.index.tolist(), 3)
+    df = timeseries_dataframe.drop(index=timestamps_to_drop)
+    missing_timestamps = _get_missing_timestamps(df, "1H")
+    assert set(missing_timestamps.index) == set(timestamps_to_drop)