Repository: dask/dask

Issue:
Keyerror when slicing dating
This looks similar to #2211 but I'm not sure. I've attached a zip with sample data and code that reproduces it. If you uncomment line 25 it works for some reason.
```
import dask.bag
import pandas as pd
import re
from datetime import datetime
schema_dict = {
    'timestamp': 'datetime64[ns]',
}


time_regex = r'\[(?P<time>[^]]+)\]'
time_regex = re.compile(time_regex)


def get_log_dict(line):
    match = time_regex.match(line)
    dt = pd.datetime.strptime(match.groupdict()['time'], '%d/%b/%Y:%H:%M:%S +0000')
    return {'timestamp': dt}


files = ['2012-09-25.log', '2012-09-26.log', '2012-09-27.log']
b = dask.bag.read_text(files, blocksize=5000000).map(get_log_dict).to_dataframe(schema_dict)
b = b[~b.timestamp.isnull()]
b = b.set_index('timestamp')
b = b[sorted(b.columns)]
# b = b.repartition(freq='15m')
start = datetime(2012, 9, 26)
end = datetime(2012, 9, 27)
b = b.loc[start:end]
b.compute()
```

[Archive.zip](https://github.com/dask/dask/files/971843/Archive.zip)



Discussion:
Yeah, this looks very similar.

I think dask's `.loc` will have to protect against the index not being monotonic / sorted, and fall back to  `boundary_slice` if it isn't. I can take a closer look tonight or tomorrow morning.
So, when you call `b.set_index('timestamp')`, it sets an index that isn't monotonic. (This is another situation where https://github.com/dask/dask/pull/2150 comes into play).

@mrocklin, @shughes-uk do you have thoughts on the desired behavior here? The two options I see:

1. Follow pandas and raise a KeyError when you do a slice into a non-monotonic index with a missing key
2. Do some special handling for this case

Normally I would go with 1 without reservation. My only hesitation is that `dask.dataframe.set_index` has that sorting behavior that differs from pandas. I think I have a slight preference for 1, but we should clarify the docs on this. The docstring for set_index mentions 

> It then passes over the data a second time, splitting up each input partition into several
pieces and sharing those pieces to all of the output partitions now in
sorted order.

when it should say mostly sorted order.

@shughes-uk you have the one work around with a repartition; the other option is to explicitly sort each partition's block before slicing:

```python
b.map_partitions(pd.DataFrame.sort_index).loc[start:end].compute()
```
Does set_index not also sort the index per-partition?  If not then should it?
> Does set_index not also sort the index per-partition? If not then should it?

Not in all cases. IIRC, we do the `partition_quantiles`, and if that is "sorted enough" (I forget the exact heuristics used), then we just set that column directly. Let me see if there's a performance cost to always mapping sort_index. Pandas does short-circut if the index is monotonic, so it may not be too expensive.
Always sorting seems reasonable to me, especially if this is the semantics expected by pandas.
If you're going to raise an error the docs should probably be changed to reflect the 'mostly sorted' status and perhaps include the workaround for it. It doesn't sound like you're going to go that route though.

Resolving patch:
diff --git a/dask/dataframe/shuffle.py b/dask/dataframe/shuffle.py
--- a/dask/dataframe/shuffle.py
+++ b/dask/dataframe/shuffle.py
@@ -79,7 +79,10 @@ def set_index(df, index, npartitions=None, shuffle=None, compute=False,
         if (mins == sorted(mins) and maxes == sorted(maxes) and
                 all(mx < mn for mx, mn in zip(maxes[:-1], mins[1:]))):
             divisions = mins + [maxes[-1]]
-            return set_sorted_index(df, index, drop=drop, divisions=divisions)
+            result = set_sorted_index(df, index, drop=drop, divisions=divisions)
+            # There are cases where this still may not be sorted
+            # so sort_index to be sure. https://github.com/dask/dask/issues/2288
+            return result.map_partitions(M.sort_index)
 
     return set_partition(df, index, divisions, shuffle=shuffle, drop=drop,
                          compute=compute, **kwargs)


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -2535,7 +2535,6 @@ def test_slice_on_filtered_boundary(drop):
     result = dd.concat([ddf, ddf.rename(columns={"B": "C"})], axis=1)
     expected = pd.concat([pdf, pdf.rename(columns={"B": "C"})], axis=1)
     assert_eq(result, expected)
-    assert not result.compute().index.is_monotonic  # didn't accidentally sort
 
 
 def test_boundary_slice_nonmonotonic():
diff --git a/dask/dataframe/tests/test_shuffle.py b/dask/dataframe/tests/test_shuffle.py
--- a/dask/dataframe/tests/test_shuffle.py
+++ b/dask/dataframe/tests/test_shuffle.py
@@ -398,6 +398,42 @@ def test_set_index_detects_sorted_data(shuffle):
     assert len(ddf2.dask) < ddf.npartitions * 4
 
 
+def test_set_index_sorts():
+    # https://github.com/dask/dask/issues/2288
+    vals = np.array([1348550149000000000, 1348550149000000000, 1348558142000000000,
+                     1348558142000000000, 1348585928000000000, 1348585928000000000,
+                     1348600739000000000, 1348601706000000000, 1348600739000000000,
+                     1348601706000000000, 1348614789000000000, 1348614789000000000,
+                     1348621037000000000, 1348621038000000000, 1348621040000000000,
+                     1348621037000000000, 1348621038000000000, 1348621040000000000,
+                     1348637628000000000, 1348638159000000000, 1348638160000000000,
+                     1348638159000000000, 1348638160000000000, 1348637628000000000,
+                     1348646354000000000, 1348646354000000000, 1348659107000000000,
+                     1348657111000000000, 1348659107000000000, 1348657111000000000,
+                     1348672876000000000, 1348672876000000000, 1348682787000000000,
+                     1348681985000000000, 1348682787000000000, 1348681985000000000,
+                     1348728167000000000, 1348728167000000000, 1348730745000000000,
+                     1348730745000000000, 1348750198000000000, 1348750198000000000,
+                     1348750198000000000, 1348753539000000000, 1348753539000000000,
+                     1348753539000000000, 1348754449000000000, 1348754449000000000,
+                     1348761333000000000, 1348761554000000000, 1348761610000000000,
+                     1348761333000000000, 1348761554000000000, 1348761610000000000,
+                     1348782624000000000, 1348782624000000000, 1348782624000000000,
+                     1348782624000000000])
+    vals = pd.to_datetime(vals, unit='ns')
+    breaks = [10, 36, 58]
+    dfs = []
+
+    for i in range(len(breaks)):
+        lo = sum(breaks[:i])
+        hi = sum(breaks[i:i + 1])
+
+        dfs.append(pd.DataFrame({"timestamp": vals[lo:hi]}, index=range(lo, hi)))
+
+    ddf = dd.concat(dfs).clear_divisions()
+    assert ddf.set_index("timestamp").index.compute().is_monotonic is True
+
+
 def test_set_index():
     dsk = {('x', 0): pd.DataFrame({'a': [1, 2, 3], 'b': [4, 2, 6]},
                                   index=[0, 1, 3]),