Repository: pyjanitor-devs/pyjanitor

Issue:
Add the ability to test and sort timestamps to be monotonic in a pandas data frame
# Brief Description

Following up on #703, this issue seeks to introduce the ability to sort the timestamps in a pandas data frame monotonically

I would like to propose...

# Example API
```python
def _test_for_monotonicity(
        df: pd.DataFrame,
        column_name: str = None,
        direction: str = 'increasing'
) -> bool:
    """
    Tests input data frame for monotonicity.

    Check if the data is monotonically increasing or decreasing.
    Direction is dependent on user input.
    Defaults to increasing

    :param df: data frame to be tested for monotonicity
    :param column_name: needs to be specified if and only if the date time is not in index.
        Defaults to None.
    :param direction: specifies the direction in which monotonicity is being tested for.
        Defaults to 'increasing'
    :return: single boolean flag indicating whether the test has passed or not
    """

def sort_monotonically(
        df: pd.DataFrame,
        column_name: str = None,
        direction: str ='increasing'
) -> pd.DataFrame:
    """
    Sorts data frame monotonically.
    It assumes the data frame has an index of type pd.DateTimeIndex when index is datetime.
    If datetime is in a column, then the column is expected to be of type pd.Timestamp

    :param df: data frame to sort monotonically
    :param column_name: needs to be specified if and only if the date time is not in index.
        Defaults to None
    :param direction: specifies the direction in which monotonicity is being tested for.
        Defaults to 'increasing'
    :return: data frame with its index sorted
    """


# more examples below
# ...
```

[ENH] Adding ability to sort timestamps monotonically
Closes #707 

# PR Description

Please describe the changes proposed in the pull request:

- Adding a function that allows monotonic sorting of timestamps in data frame 

**This PR resolves #707 **

# PR Checklist
Please ensure that you have done the following:

1. [x] PR in from a fork off your branch. Do not PR from `<your_username>`:`dev`, but rather from `<your_username>`:`<feature-branch_name>`.
2. [x] If you're not on the contributors list, add yourself to `AUTHORS.rst`.
3. [x] Add a line to `CHANGELOG.rst` under the latest version header (i.e. the one that is "on deck") describing the contribution.
    - Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

## Quick Check

To do a very quick check that everything is correct, follow these steps below:

- [x] Run the command `make check` from pyjanitor's top-level directory. This will automatically run:
    - black formatting
    - flake8 checking
    - running the test suite
    - docs build

Once done, please check off the check-box above.

If `make check` does not work for you, you can execute the commands listed in the Makefile individually.

## Code Changes

<!-- If you have not made code changes, please feel free to delete this section. -->

If you are adding code changes, please ensure the following:

- [x] Ensure that you have added tests.
- [x] Run all tests (`$ pytest .`) locally on your machine.
    - [x] Check to ensure that test coverage covers the lines of code that you have added.
    - [x] Ensure that all tests pass.

## Documentation Changes
If you are adding documentation changes, please ensure the following:

- [x] Build the docs locally.
- [ ] View the docs to check that it renders correctly.

# Relevant Reviewers
- @ericmjl 
- @samukweku 

Please tag maintainers to review.

- @ericmjl



Discussion:
```python 
def _test_for_monotonicity(
        df: pd.DataFrame,
        column_name: str = None,
        direction: str = 'increasing'
) -> bool:
    """
    Tests input data frame for monotonicity.

    Check if the data is monotonically increasing or decreasing.
    Direction is dependent on user input.
    Defaults to increasing

    :param df: data frame to be tested for monotonicity
    :param column_name: needs to be specified if and only if the date time is not in index.
        Defaults to None.
    :param direction: specifies the direction in which monotonicity is being tested for.
        Defaults to 'increasing'
    :return: single boolean flag indicating whether the test has passed or not
    """
    # Check all the inputs are the correct data type
    check("df", df, [pd.DataFrame])
    check("column_name", column_name, [str, None])
    check("direction", direction, [str])

    # Test whether the column is pd.Timestamp
    df1 = df.copy(deep=True)
    if column_name:
        df1.index.name = "idx"
        check("idx", df1.index, [pd.Timestamp]), "column_name should be pandas timestamp"
        df1 = df1.reset_index()
    else:
        index_in_column = df1[[column_name]]
        check("index_in_column", index_in_column, [pd.Timestamp]),  "column_name should be pandas timestamp"
        df1 = df1.rename(columns={column_name: "idx"}, inplace=True)

    # Test for monotonicity
    if direction == 'increasing':
        return df1["idx"].is_monotonic()
    elif direction == 'decreasing':
        return df1["idx"].is_monotonic_decreasing()
    else:
        raise ValueError("\n Argument direction accepts only increasing or decreasing ")


def sort_monotonically(
        df: pd.DataFrame,
        column_name: str = None,
        direction: str ='increasing'
) -> pd.DataFrame:
    """
    Sorts data frame monotonically.
    It assumes the data frame has an index of type pd.DateTimeIndex when index is datetime.
    If datetime is in a column, then the column is expected to be of type pd.Timestamp

    :param df: data frame to sort monotonically
    :param column_name: needs to be specified if and only if the date time is not in index.
        Defaults to None
    :param direction: specifies the direction in which monotonicity is being tested for.
        Defaults to 'increasing'
    :return: data frame with its index sorted
    """
    monotonic_flag = _test_for_monotonicity(
        df,
        column_name,
        direction
    )

    # Standardize the data frame
    if column_name:
        df = df.set_index(column_name)

    if monotonic_flag:
        return df
    else:
        if direction == "increasing":
            df = df.sort_index()
        else:
            df = df.sort_index(ascending=False)
        return df

```

I am waiting for closure on https://github.com/ericmjl/pyjanitor/issues/705 , I have already developed the code to attempt these 
@ericmjl , 
       I am ready to do a pull request to address this, however, when I try it is pulling in changes from the previous issue as well. I made sure to do a ``` git pull ``` before working on the issue. So, not sure what I am doing wrong. Any advice ? 
Hey @UGuntupalli! I'm sorry for getting back late here. (Had a bit of a rough end to the week, took some time to rest.)

Have you tried deleting your fork and re-forking, then re-opening up the repo in a dev container? That will clear out any state and give you a clean slate to start from.


Resolving patch:
diff --git a/janitor/timeseries.py b/janitor/timeseries.py
--- a/janitor/timeseries.py
+++ b/janitor/timeseries.py
@@ -87,3 +87,58 @@ def _get_missing_timestamps(
     missing_timestamps = expected_df.index.difference(df.index)
 
     return expected_df.loc[missing_timestamps]
+
+
+@pf.register_dataframe_method
+def sort_timestamps_monotonically(
+    df: pd.DataFrame, direction: str = "increasing", strict: bool = False
+) -> pd.DataFrame:
+    """
+    Sort dataframe such that index is monotonic.
+
+    If timestamps are monotonic,
+    this function will return the dataframe unmodified.
+    If timestamps are not monotonic,
+    then the function will sort the dataframe.
+
+    Example usage:
+
+    .. code-block:: python
+
+        df = (
+            pd.DataFrame(...)
+            .sort_timestamps_monotonically(direction='increasing')
+        )
+
+    :param df: Dataframe which needs to be tested for monotonicity
+    :param direction: type of monotonicity desired.
+        Acceptable arguments are:
+            1. increasing
+            2. decreasing
+    :param strict: flag to enable/disable strict monotonicity.
+        If set to True,
+        will remove duplicates in the index,
+        by retaining first occurrence of value in index.
+        If set to False,
+        will not test for duplicates in the index.
+        Defaults to False.
+    :returns: Dataframe that has monotonically increasing
+        (or decreasing) timestamps.
+    """
+    # Check all the inputs are the correct data type
+    check("df", df, [pd.DataFrame])
+    check("direction", direction, [str])
+    check("strict", strict, [bool])
+
+    # Remove duplicates if requested
+    if strict:
+        df = df[~df.index.duplicated(keep="first")]
+
+    # Sort timestamps
+    if direction == "increasing":
+        df = df.sort_index()
+    else:
+        df = df.sort_index(ascending=False)
+
+    # Return the dataframe
+    return df


Test patch:
diff --git a/tests/timeseries/test_sort_timestamps_monotonically.py b/tests/timeseries/test_sort_timestamps_monotonically.py
new file mode 100644
--- /dev/null
+++ b/tests/timeseries/test_sort_timestamps_monotonically.py
@@ -0,0 +1,52 @@
+import pytest
+import pandas as pd
+from random import randint
+import janitor  # noqa: F401
+import janitor.timeseries  # noqa: F401
+
+
+@pytest.fixture
+def timeseries_dataframe() -> pd.DataFrame:
+    """
+    Returns a time series dataframe
+    """
+    ts_index = pd.date_range("1/1/2019", periods=1000, freq="1H")
+    v1 = [randint(1, 2000) for i in range(1000)]
+    test_df = pd.DataFrame({"v1": v1}, index=ts_index)
+    return test_df
+
+
+# NOTE: The tests possibly can be merged back together later
+# if they are parametrized properly.
+# NOTE: These tests use `df.equals(other_df)`,
+# because the desired `pd.assert_frame_equal(df, other_df)`
+# constantly failed on the CI systems.
+# It's a task for later to fix.
+
+
+@pytest.mark.timeseries
+def test_sort_timestamps_monotonically(timeseries_dataframe):
+    """Test sort_timestamps_monotonically for ascending order"""
+    df = timeseries_dataframe.shuffle(
+        reset_index=False
+    ).sort_timestamps_monotonically()
+    assert df.equals(timeseries_dataframe)
+
+
+@pytest.mark.timeseries
+def test_sort_timestamps_monotonically_decreasing(timeseries_dataframe):
+    """Test sort_timestamps_monotonically for descending order"""
+    df2 = timeseries_dataframe.sort_index(ascending=False)
+    df3 = df2.sort_timestamps_monotonically("decreasing")
+    assert df3.equals(df2)
+
+
+@pytest.mark.timeseries
+def test_sort_timestamps_monotonically_strict(timeseries_dataframe):
+    """Test sort_timestamps_monotonically for index duplication handling"""
+    df = timeseries_dataframe.shuffle(reset_index=False)
+    random_number = randint(1, len(timeseries_dataframe))
+    df = df.append(
+        df.loc[df.index[random_number], :]
+    ).sort_timestamps_monotonically(direction="increasing", strict=True)
+    assert df.equals(timeseries_dataframe)