Repository: ericmjl/pyjanitor (python)

Issue:
Add the ability to flag and fill if timestamps are missing in a pandas data frame
# Brief Description

Following up on https://github.com/ericmjl/pyjanitor/issues/703, this issue seeks to introduce the ability to flag if timestamps are missing in a given data frame, provided the frequency of the time series data in the data frame  

I would like to propose...

# Example API
```python
import pandas as pd 
import janitor 

def _flag_missing_timestamps(df, frequency, first_time_stamp=None, last_time_stamp=None):
    """
    Utility function to test that if input time_series is missing any timestamps relative to expected timestamps
    generated based on the first_time_stamp, last_time_stamp and frequency.

    :param pd.DataFrame df: data frame which needs to be tested for missing timestamps
    :param int frequency: frequency i.e. sampling frequency of the data, expressed in seconds. For e.g. if the data has
    hourly frequency, then a value of 60 * 60 = 3600 is expected
    :param pd.Timestamp first_time_stamp: timestamp at which the time_series is expected to start from. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :param pd.Timestamp last_time_stamp: timestamp at which the time_series is expected to end with. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :return: boolean flag indicating whether there are any missing timestamps in the input time_series
    :rtype: bool
    """

@pf.register_dataframe_method
def fill_missing_timestamps(df, frequency, column_name=None, first_time_stamp=None, last_time_stamp=None): 
    """
    Re-index the data frame if there are missing timestamps found in the data frame.  
    
    :param pd.DataFrame df: 
    :param int frequency: frequency i.e. sampling frequency of the data, expressed in seconds. For e.g. if the data has
    hourly frequency, then a value of 60 * 60 = 3600 is expected
    :param str column_name: name of the column which has time series if not the index. Defaults to None. By default, the 
    index is used for checking for the timestamps, unless a value other than None is assigned to column_name 
    :param pd.Timestamp first_time_stamp: timestamp at which the time_series is expected to start from. Defaults to None
    If no input is provided, assumes the minimum value in time_series
    :param pd.Timestamp last_time_stamp: timestamp at which the time_series is expected to end with. Defaults to None
    If no input is provided, assumes the minimum value in time_series 
    :return: reindexed data frame if it turns out to have missing timestamps, else returns the input data frame as is 
    :rtype: pd.DataFrame
    """

```


Pull request:
[ENH] first pass on fill_missing_timestamps
Closes #705

# PR Description

Please describe the changes proposed in the pull request:

- Introducing a "timeseries.py" module with data checking and data cleaning functions geared towards time series data 
- Adding two functions (a utility function and an actual user-facing function) to fill_missing_timestamps in a timeseries data set 

**This PR resolves #705.**

# PR Checklist

Please ensure that you have done the following:

1. [x] PR in from a fork off your branch. Do not PR from `<your_username>`:`dev`, but rather from `<your_username>`:`<feature-branch_name>`. 
         I think I am doing this, but I am really not sure, so please guide me through and I will be happy to resubmit. 

2. [x] If you're not on the contributors list, add yourself to `AUTHORS.rst`.

3. [x] Add a line to `CHANGELOG.rst` under the latest version header (i.e. the one that is "on deck") describing the contribution.
    - Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

## Quick Check
**_I have made code and documentation changes, however I don't have enough experience in adding unit tests. I just added one unit test using an existing example from the code base, but I need some help with ```pytest```. I have used unittest very briefly in the past, but any guidance regarding the testing piece will be helpful for me to progress further._**  


To do a very quick check that everything is correct, follow these steps below:

- [x] Run the command `make check` from pyjanitor's top-level directory. This will automatically run:
    - black formatting
    - flake8 checking
    - running the test suite
    - docs build

```
tests\conftest.py:7: in <module>
    from janitor.testing_utils import date_data
E   ModuleNotFoundError: No module named 'janitor'
```

## Code Changes

If you are adding code changes, please ensure the following:

- [ ] Ensure that you have added tests.
- [ ] Run all tests (`$ pytest .`) locally on your machine.
    - [ ] Check to ensure that test coverage covers the lines of code that you have added.
    - [ ] Ensure that all tests pass.

## Documentation Changes

<!-- If you have not made documentation changes, please feel free to delete this section. -->

If you are adding documentation changes, please ensure the following:

- [ ] Build the docs locally.
- [ ] View the docs to check that it renders correctly.

# Relevant Reviewers

<!-- Finally, please tag relevant maintainers to review. -->

Please tag maintainers to review.

- @ericmjl

Patch:
diff --git a/AUTHORS.rst b/AUTHORS.rst
index b8b461b..ca89d46 100644
--- a/AUTHORS.rst
+++ b/AUTHORS.rst
@@ -90,4 +90,5 @@ Contributors
 - `@evan-anderson <https://github.com/evan-anderson>`_ | `contributions <https://github.com/ericmjl/pyjanitor/pulls?utf8=%E2%9C%93&q=is%3Aclosed+mentions%3evan-anderson>`_
 - `@smu095 <https://github.com/smu095>`_ | `contributions <https://github.com/ericmjl/pyjanitor/issues?q=is%3Aclosed+mentions%3smu095>`_
 - `@VPerrollaz <https://github.com/VPerrollaz>`_ | `contributions <https://github.com/ericmjl/pyjanitor/issues?q=is%3Aclosed+mentions%3AVPerrollaz>`_
+- `@UGuntupalli <https://github.com/UGuntupalli>`_ | `contributions < https://github.com/ericmjl/pyjanitor/issues?q=is%3Aclosed+mentions%3AUGuntupalli >`_
 - `@mphirke <https://github.com/mphirke>`_ | `contributions <https://github.com/ericmjl/pyjanitor/issues?q=is%3Aclosed+mentions%3Amphirke>`_
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index ffc84ca..4f258d3 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -23,6 +23,7 @@ v0.20.8
 - [ENH] Added ``fill_direction`` function for forward/backward fills on missing values
   for selected columns in a dataframe. @samukweku
 - [ENH] Simpler logic and less lines of code for expand_grid function @samukweku
+ 
 
 
 v0.20.7
diff --git a/janitor/timeseries.py b/janitor/timeseries.py
new file mode 100644
index 0000000..5e1a99d
--- /dev/null
+++ b/janitor/timeseries.py
@@ -0,0 +1,89 @@
+"""
+Time series-specific data testing and cleaning functions.
+"""
+
+import pandas as pd
+import pandas_flavor as pf
+from janitor import check
+
+
+@pf.register_dataframe_method
+def fill_missing_timestamps(
+    df: pd.DataFrame,
+    frequency: str,
+    first_time_stamp: pd.Timestamp = None,
+    last_time_stamp: pd.Timestamp = None,
+) -> pd.DataFrame:
+    """
+    Fill dataframe with missing timestamps based on a defined frequency.
+
+    If timestamps are missing,
+    this function will reindex the dataframe.
+    If timestamps are not missing,
+    then the function will return the dataframe unmodified.
+    Example usage:
+    .. code-block:: python
+
+        df = (
+            pd.DataFrame(...)
+            .fill_missing_timestamps(frequency="1H")
+        )
+
+    :param df: Dataframe which needs to be tested for missing timestamps
+    :param frequency: frequency i.e. sampling frequency of the data.
+        Acceptable frequency strings are available
+        `here <https://pandas.pydata.org/pandas-docs/stable/>`_
+        Check offset aliases under time series in user guide
+    :param first_time_stamp: timestamp expected to start from
+        Defaults to None.
+        If no input is provided assumes the minimum value in time_series
+    :param last_time_stamp: timestamp expected to end with.
+        Defaults to None.
+        If no input is provided, assumes the maximum value in time_series
+    :returns: dataframe that has a complete set of contiguous datetimes.
+    """
+    # Check all the inputs are the correct data type
+    check("frequency", frequency, [str])
+    check("first_time_stamp", first_time_stamp, [pd.Timestamp, type(None)])
+    check("last_time_stamp", last_time_stamp, [pd.Timestamp, type(None)])
+
+    if first_time_stamp is None:
+        first_time_stamp = df.index.min()
+    if last_time_stamp is None:
+        last_time_stamp = df.index.max()
+
+    # Generate expected timestamps
+    expected_timestamps = pd.date_range(
+        start=first_time_stamp, end=last_time_stamp, freq=frequency
+    )
+
+    return df.reindex(expected_timestamps)
+
+
+def _get_missing_timestamps(
+    df: pd.DataFrame,
+    frequency: str,
+    first_time_stamp: pd.Timestamp = None,
+    last_time_stamp: pd.Timestamp = None,
+) -> pd.DataFrame:
+    """
+    Return the timestamps that are missing in a dataframe.
+
+    This function takes in a dataframe,
+    and checks its index against a dataframe
+    that contains the expected timestamps.
+    Here, we assume that the expected timestamps
+    are going to be of a larger size
+    than the timestamps available in the input dataframe ``df``.
+
+    If there are any missing timestamps in the input dataframe,
+    this function will return those missing timestamps
+    from the expected dataframe.
+    """
+    expected_df = df.fill_missing_timestamps(
+        frequency, first_time_stamp, last_time_stamp
+    )
+
+    missing_timestamps = expected_df.index.difference(df.index)
+
+    return expected_df.loc[missing_timestamps]


Test patch:
diff --git a/tests/timeseries/test_fill_missing_timestamps.py b/tests/timeseries/test_fill_missing_timestamps.py
new file mode 100644
index 0000000..055294d
--- /dev/null
+++ b/tests/timeseries/test_fill_missing_timestamps.py
@@ -0,0 +1,48 @@
+import pytest
+import pandas as pd
+from random import randint
+from janitor.timeseries import fill_missing_timestamps, _get_missing_timestamps
+
+
+# Random data for testing
+@pytest.fixture
+def timeseries_dataframe() -> pd.DataFrame:
+    """
+    Returns a time series dataframe
+    """
+    ts_index = pd.date_range("1/1/2019", periods=1000, freq="1H")
+    v1 = [randint(1, 2000) for i in range(1000)]
+    test_df = pd.DataFrame({"v1": v1}, index=ts_index)
+    return test_df
+
+
+@pytest.mark.timeseries
+def test_fill_missing_timestamps(timeseries_dataframe):
+    """Test that filling missing timestamps works as expected."""
+    # Remove random row from the data frame
+    random_number = randint(1, len(timeseries_dataframe))
+    df1 = timeseries_dataframe.drop(timeseries_dataframe.index[random_number])
+
+    # Fill missing timestamps
+    result = fill_missing_timestamps(df1, frequency="1H")
+
+    # Testing if the missing timestamp has been filled
+    assert len(result) == len(timeseries_dataframe)
+
+    # Testing if indices are exactly the same after filling
+    original_index = timeseries_dataframe.index
+    new_index = result.index
+    delta = original_index.difference(new_index)
+
+    assert delta.empty is True
+
+
+@pytest.mark.timeseries
+def test__get_missing_timestamps(timeseries_dataframe):
+    """Test utility function for identifying the missing timestamps."""
+    from random import sample
+
+    timestamps_to_drop = sample(timeseries_dataframe.index.tolist(), 3)
+    df = timeseries_dataframe.drop(index=timestamps_to_drop)
+    missing_timestamps = _get_missing_timestamps(df, "1H")
+    assert set(missing_timestamps.index) == set(timestamps_to_drop)