Repository: dask/dask

Issue:
set_index(sorted=True) fails with an empty frame
**What happened**: when trying to set an index on an empty frame, it works fine with `set_index(sorted=False)` -except for `divisions` being set to `nan`-, but it crashes with `set_index(sorted=True)`.

**What you expected to happen**: given that it's empty, it should yield the same result. It's a bit ambiguous whether an empty frame's index is "sorted" or not (see discussion below), but certainly it shouldn't crash like this.

**Minimal Complete Verifiable Example**:

```python
>>> import dask.dataframe as dd
>>> import pandas as pd

>>> ddf = dd.from_pandas(pd.DataFrame({'a': [], 'b': []}), npartitions=1)
>>> ddf.compute()
Empty DataFrame
Columns: [a, b]
Index: []
>>> ddf.divisions
(None, None)

>>> ddf.set_index('a').compute()
Empty DataFrame
Columns: [b]
Index: []
>>> ddf.set_index('a').divisions
(nan, nan)

>>> ddf.set_index('a', sorted=True).compute()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jorge.lopez/anaconda3/envs/AML/lib/python3.9/site-packages/dask/dataframe/core.py", line 4689, in set_index
    return set_sorted_index(
  File "/home/jorge.lopez/anaconda3/envs/AML/lib/python3.9/site-packages/dask/dataframe/shuffle.py", line 1092, in set_sorted_index
    return compute_and_set_divisions(result, **kwargs)
  File "/home/jorge.lopez/anaconda3/envs/AML/lib/python3.9/site-packages/dask/dataframe/shuffle.py", line 1072, in compute_and_set_divisions
    return fix_overlap(df, mins, maxes, lens)
  File "/home/jorge.lopez/anaconda3/envs/AML/lib/python3.9/site-packages/dask/dataframe/shuffle.py", line 990, in fix_overlap
    divisions = tuple(mins) + (maxes[-1],)
IndexError: list index out of range
```

**Anything else we need to know?**: 

This is probably a side effect of #8517 , which made some changes in `fix_overlap`, the method that eventually fails. The difference between `set_index(sorted=False)` and `set_index(sorted=True)` is that the latter calls `set_sorted_index` and `compute_and_set_divisions`, as can be seen in the trace above.

IMHO, and following a [discussion about divisions](https://github.com/dask/dask/issues/8845#issuecomment-1081584276) in #8845, **when a dataframe is empty then divisions should always be cleared, as we have no idea which are the `mins` and the `maxes` and the concept of `divisions` themselves doesn't make much sense**. 

This is what makes the code fail in `fix_overlap` while trying to access the first position of the `mins` and `maxes`, when that position doesn't even exist. The case with `sorted=False` works, but note that `divisions` become `(nan, nan)` instead of `(None, None)`, which is probably not what we want!

Note also that this fact makes the concept of "sortedness" ambiguous, but for the particular case of a no-rows dataframe, I think it should be considered as sorted (same as in Python `all([])` is `True`), and do nothing but:

1. Move the requested column to be the index.
2. Clear `divisions`, setting all of them to `None` (there's nothing to calculate, no quantiles to consider, etc).

**Environment**:

- Dask version: 2022.04.0
- Python version: 3.9.12
- Operating System: Ubuntu
- Install method (conda, pip, source): Pip


Discussion:
Thanks for opening this @jorloplaz I will take a look.

Resolving patch:
diff --git a/dask/dataframe/shuffle.py b/dask/dataframe/shuffle.py
--- a/dask/dataframe/shuffle.py
+++ b/dask/dataframe/shuffle.py
@@ -999,6 +999,13 @@ def fix_overlap(ddf, mins, maxes, lens):
     name = "fix-overlap-" + tokenize(ddf, mins, maxes, lens)
 
     non_empties = [i for i, length in enumerate(lens) if length != 0]
+    # If all empty, collapse into one partition
+    if len(non_empties) == 0:
+        divisions = (None, None)
+        dsk = {(name, 0): (ddf._name, 0)}
+        graph = HighLevelGraph.from_collections(name, dsk, dependencies=[ddf])
+        return new_dd_object(graph, name, ddf._meta, divisions)
+
     # drop empty partitions by mapping each partition in a new graph to a particular
     # partition on the old graph.
     dsk = {(name, i): (ddf._name, div) for i, div in enumerate(non_empties)}


Test patch:
diff --git a/dask/dataframe/tests/test_shuffle.py b/dask/dataframe/tests/test_shuffle.py
--- a/dask/dataframe/tests/test_shuffle.py
+++ b/dask/dataframe/tests/test_shuffle.py
@@ -878,7 +878,6 @@ def test_set_index_sorted_min_max_same():
 
 def test_set_index_empty_partition():
     test_vals = [1, 2, 3]
-
     converters = [int, float, str, lambda x: pd.to_datetime(x, unit="ns")]
 
     for conv in converters:
@@ -906,11 +905,17 @@ def test_set_index_on_empty():
 
         assert ddf.npartitions > 1
 
-        ddf = ddf[ddf.y > df.y.max()].set_index("x")
-        expected_df = df[df.y > df.y.max()].set_index("x")
+        actual = ddf[ddf.y > df.y.max()].set_index("x")
+        expected = df[df.y > df.y.max()].set_index("x")
+
+        assert assert_eq(actual, expected, **CHECK_FREQ)
+        assert actual.npartitions == 1
+        assert all(pd.isnull(d) for d in actual.divisions)
 
-        assert assert_eq(ddf, expected_df, **CHECK_FREQ)
-        assert ddf.npartitions == 1
+        actual = ddf[ddf.y > df.y.max()].set_index("x", sorted=True)
+        assert assert_eq(actual, expected, **CHECK_FREQ)
+        assert actual.npartitions == 1
+        assert all(pd.isnull(d) for d in actual.divisions)
 
 
 def test_set_index_categorical():