Repository: dask/dask

Issue:
is_monotonic_increasing (resp. decreasing) still fails in some cases
**What happened**: after merging #8897, `is_monotonic_increasing` still fails in some circumstances, throwing a surprising error about `first` and `last` columns not being there.

**What you expected to happen**: shouldn't fail at all.

**Minimal Complete Verifiable Example**:

<details>
<summary>Code</summary>

```python
# imports
import pandas as pd
import dask.dataframe as dd
from dask import delayed

# simple function to create one partition for each date
@delayed
def create_partition(date, values):
     index = pd.DatetimeIndex([date] * len(values), name="date")
     frame = pd.DataFrame({"foo": values}, index=index)
     return frame

# fix dates (8 days) and values
dates = pd.date_range("2020-01-01", "2020-01-08", freq="D")
values = [1, 2, 3]

# this works fine and yields True
ddf = dd.from_delayed(dfs=[create_partition(date, values) for date in dates], divisions=list(dates)+[dates[-1]], verify_meta=False)
print(ddf.index.is_monotonic_increasing.compute())

# however, with 9 days the same code crashes!!!
dates = pd.date_range("2020-01-01", "2020-01-09", freq="D")     # 9 instead of 8
ddf = dd.from_delayed(dfs=[create_partition(date, values) for date in dates], divisions=list(dates)+[dates[-1]], verify_meta=False)
print(ddf.index.is_monotonic_increasing.compute())
```
</details>

<details>
<summary>Full error trace</summary>

```python
>>> print(ddf.index.is_monotonic_increasing.compute())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/dask/base.py", line 292, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/dask/base.py", line 575, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/assuming `split_every=8` if called with `None`python3.9/site-packages/dask/threaded.py", line 81, in get
    results = get_async(
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/dask/local.py", line 508, in get_async
    raise_exception(exc, tb)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/dask/local.py", line 221, in execute_task
    result = _execute_task(task, data)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/toolz/functoolz.py", line 630, in pipe
    data = func(data)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/dask/dataframe/methods.py", line 457, in monotonic_increasing_aggregate
    concatenated[["first", "last"]].to_numpy().ravel()
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/series.py", line 984, in __getitem__
    return self._get_with(key)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/series.py", line 1024, in _get_with
    return self.loc[key]
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 967, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 1191, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 1132, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 1327, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 5782, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/jorge.lopez/anaconda3/envs/AML-dev/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 5842, in _raise_if_missingassuming `split_every=8` if called with `None`

    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['first', 'last'], dtype='object')] are in the [index]"
```
</details>

**Anything else we need to know?**: the striking fact that it fails with 9 days but not with 8 (or less) points to the [aca method assuming `split_every=8` if called with `split_every=None`](https://github.com/dask/dask/blob/b0e234f65577a35ae4d78e0569ee3f44235f6089/dask/dataframe/core.py#L6100), which is [what `is_monotonic_increasing` does](https://github.com/dask/dask/blob/b0e234f65577a35ae4d78e0569ee3f44235f6089/dask/dataframe/core.py#L3999). 

The [aggregating function `monotonic_increasing_aggregate`](https://github.com/dask/dask/blob/b0e234f65577a35ae4d78e0569ee3f44235f6089/dask/dataframe/methods.py#L457) assumes that columns  `first` and `last` have been obtained by the [chunking function `monotonic_increasing_chunk`](https://github.com/dask/dask/blob/b0e234f65577a35ae4d78e0569ee3f44235f6089/dask/dataframe/methods.py#L452). However, it seems that splitting is ruining this somehow... 

**Note this doesn't have to do with empty partitions, as `create_partition` is creating non-empty partitions for all dates.**

**Same comments are applicable for `is_monotonic_decreasing`.**

**Environment**:

- Dask version: 2022.04.1
- Python version: 3.9.12
- Operating System: Ubuntu
- Install method (conda, pip, source): Conda


Discussion:
Thanks for opening this @jorloplaz! As always, if you have the capacity to work on this a PR would be appreciated :)
If you could give me some hints on why `is_monotonic()` seems to be implemented as an ad-hoc reduction (with its own `chunk` and `aggregate` functions), I could give it a try. 

My understanding, as it is already implemented in Pandas, is that it should be very similar to other Pandas reductions. So basically delegating on it, something like `series.map_partitions(M.is_monotonic).all()` as a first check. 

And after that, it's just checking that the first element of each partition is `>=` the last element of the previous one. So probably with a `map_overlap` it could be done just going once through all data?
Yeah that's a good point. I bet you could do it with a `map_overlap` but it might be less performant since each task will require 3 partitions as input rather than one. I just tried to implement it using `map_overlap` and got a recursion error so there might be some other reason that I am missing. 

In terms of how to solve the issue at hand I think that it would require another custom function. Right now there is a chunk function and an aggregate function but there is no combine function. When you get above a certain number of partitions dask starts doing tree aggregations so the combine function would operate on the concatenated outputs from the aggregate function and do get the result.
In writing this up I actually make a little PR. But please feel free to investigate the map_overlap approach if you feel so inclined.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -4011,6 +4011,7 @@ def is_monotonic_increasing(self):
         return aca(
             self,
             chunk=methods.monotonic_increasing_chunk,
+            combine=methods.monotonic_increasing_combine,
             aggregate=methods.monotonic_increasing_aggregate,
             meta=bool,
             token="monotonic_increasing",
@@ -4022,6 +4023,7 @@ def is_monotonic_decreasing(self):
         return aca(
             self,
             chunk=methods.monotonic_decreasing_chunk,
+            combine=methods.monotonic_decreasing_combine,
             aggregate=methods.monotonic_decreasing_aggregate,
             meta=bool,
             token="monotonic_decreasing",
diff --git a/dask/dataframe/methods.py b/dask/dataframe/methods.py
--- a/dask/dataframe/methods.py
+++ b/dask/dataframe/methods.py
@@ -1,4 +1,5 @@
 import warnings
+from functools import partial
 
 import numpy as np
 import pandas as pd
@@ -442,35 +443,44 @@ def assign_index(df, ind):
     return df
 
 
-def monotonic_increasing_chunk(x):
+def _monotonic_chunk(x, prop):
     if x.empty:
         # if input is empty, return empty df for chunk
         data = None
     else:
         data = x if is_index_like(x) else x.iloc
-        data = [[x.is_monotonic_increasing, data[0], data[-1]]]
+        data = [[getattr(x, prop), data[0], data[-1]]]
     return pd.DataFrame(data=data, columns=["monotonic", "first", "last"])
 
 
-def monotonic_increasing_aggregate(concatenated):
-    bounds_are_monotonic = pd.Series(
-        concatenated[["first", "last"]].to_numpy().ravel()
-    ).is_monotonic_increasing
-    return concatenated["monotonic"].all() and bounds_are_monotonic
-
-
-def monotonic_decreasing_chunk(x):
-    if x.empty:
-        # if input is empty, return empty df for chunk
+def _monotonic_combine(concatenated, prop):
+    if concatenated.empty:
         data = None
     else:
-        data = x if is_index_like(x) else x.iloc
-        data = [[x.is_monotonic_decreasing, data[0], data[-1]]]
-    return pd.DataFrame(data=data, columns=["monotonic", "first", "last"])
+        s = pd.Series(concatenated[["first", "last"]].to_numpy().ravel())
+        is_monotonic = concatenated["monotonic"].all() and getattr(s, prop)
+        data = [[is_monotonic, s.iloc[0], s.iloc[-1]]]
+    return pd.DataFrame(data, columns=["monotonic", "first", "last"])
+
 
+def _monotonic_aggregate(concatenated, prop):
+    s = pd.Series(concatenated[["first", "last"]].to_numpy().ravel())
+    return concatenated["monotonic"].all() and getattr(s, prop)
 
-def monotonic_decreasing_aggregate(concatenated):
-    bounds_are_monotonic = pd.Series(
-        concatenated[["first", "last"]].to_numpy().ravel()
-    ).is_monotonic_decreasing
-    return concatenated["monotonic"].all() and bounds_are_monotonic
+
+monotonic_increasing_chunk = partial(_monotonic_chunk, prop="is_monotonic_increasing")
+monotonic_decreasing_chunk = partial(_monotonic_chunk, prop="is_monotonic_decreasing")
+
+monotonic_increasing_combine = partial(
+    _monotonic_combine, prop="is_monotonic_increasing"
+)
+monotonic_decreasing_combine = partial(
+    _monotonic_combine, prop="is_monotonic_decreasing"
+)
+
+monotonic_increasing_aggregate = partial(
+    _monotonic_aggregate, prop="is_monotonic_increasing"
+)
+monotonic_decreasing_aggregate = partial(
+    _monotonic_aggregate, prop="is_monotonic_decreasing"
+)


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -5113,12 +5113,12 @@ def test_index_is_monotonic_numeric():
 
 
 def test_index_is_monotonic_dt64():
-    s = pd.Series(1, index=pd.date_range("20130101", periods=10))
-    ds = dd.from_pandas(s, npartitions=5, sort=False)
+    s = pd.Series(1, index=pd.date_range("20130101", periods=20))
+    ds = dd.from_pandas(s, npartitions=10, sort=False)
     assert_eq(s.index.is_monotonic_increasing, ds.index.is_monotonic_increasing)
 
     s_2 = pd.Series(1, index=list(reversed(s)))
-    ds_2 = dd.from_pandas(s_2, npartitions=5, sort=False)
+    ds_2 = dd.from_pandas(s_2, npartitions=10, sort=False)
     assert_eq(s_2.index.is_monotonic_decreasing, ds_2.index.is_monotonic_decreasing)