Repository: dask/dask

Issue:
set_index on datetime column with microsecond precision removes one row from dataframe
**What happened**:

I created a pandas dataframe with shape (4, 2). One of the column is a datetime with microsecond precision.

If I create a dask dataframe from this pandas dataframe and call `set_index` on the datetime column, the resulting dataframe now has only 3 rows:
![image](https://user-images.githubusercontent.com/25746704/99700936-d180b700-2a61-11eb-966f-c534228cd26f.png)


**What you expected to happen**:

If I comment out the line where we convert the index column to a datetime, then I get a dataframe with 4 rows:
![image](https://user-images.githubusercontent.com/25746704/99701099-012fbf00-2a62-11eb-833c-1aa07b30e545.png)

I expected the original code snippet to produce the same output (i.e. 4 row dataframe), but with the index as a datetime instead of a number of microseconds since 1970-01-01. 

As a workaround, if I convert the column to datetime **after setting the index**, then I get what I really want:
![image](https://user-images.githubusercontent.com/25746704/99703034-9d5ac580-2a64-11eb-97d5-910c573a4fa1.png)


**Minimal Complete Verifiable Example**:

```python
import dask.dataframe as dd
import pandas as pd

df = pd.DataFrame(
    [
        [1567703791155681, 1],
        [1567703792155681, 2],
        [1567703790155681, 0],
        [1567703793155681, 3],
    ],
    columns=["ts", "rank"]
)
df.ts = pd.to_datetime(df.ts, unit='us')  # comment this line to get a df with 4 lines

ddf = dd.from_pandas(df, npartitions=2)

ddf = ddf.set_index("ts")

ddf.compute()
```


**Anything else we need to know?**:
I suspect the problem comes from the fact that the divisions of the dask dataframe after `set_index` are as follows:
![image](https://user-images.githubusercontent.com/25746704/99702095-63d58a80-2a63-11eb-8297-87d579029c7a.png)

Notice that the divisions are precise to the nanosecond and that the first division is bigger than the timestamp of the row we've lost by 24 nanoseconds... Looks like numerical imprecision microseconds-to-datetime conversion maybe?


**Environment**:

- Dask version: 2.30.0
- Python version: 3.8.6
- Operating System: Linux (FROM python:3.8.6-buster in Dockerfile)
- Install method (conda, pip, source): pip

n.b. I ran the above snippet in a jupyter notebook.


Discussion:
Thanks for the nice reproducer @dtourillon !

I was able to reproduce the problem and tracked the precision error to [the round-trip casting](https://github.com/dask/dask/blob/ff3ea6c74e8736a5823d648defcf9164f083d266/dask/array/percentile.py#L28) used to enable the `np.percentile` call on each partition of the index.  One possible fix is to check (after the conversion) that the first division is no greater than the minimum value in the index.  Or perhaps we could always set the first division to the minimum...

Resolving patch:
diff --git a/dask/array/percentile.py b/dask/array/percentile.py
--- a/dask/array/percentile.py
+++ b/dask/array/percentile.py
@@ -24,8 +24,11 @@ def _percentile(a, q, interpolation="linear"):
         return pd.Categorical.from_codes(result, a.categories, a.ordered), n
     if np.issubdtype(a.dtype, np.datetime64):
         a2 = a.astype("i8")
-        result = np.percentile(a2, q, interpolation=interpolation)
-        return result.astype(a.dtype), n
+        result = np.percentile(a2, q, interpolation=interpolation).astype(a.dtype)
+        if q[0] == 0:
+            # https://github.com/dask/dask/issues/6864
+            result[0] = min(result[0], a.min())
+        return result, n
     if not np.issubdtype(a.dtype, np.number):
         interpolation = "nearest"
     return np.percentile(a, q, interpolation=interpolation), n


Test patch:
diff --git a/dask/dataframe/tests/test_shuffle.py b/dask/dataframe/tests/test_shuffle.py
--- a/dask/dataframe/tests/test_shuffle.py
+++ b/dask/dataframe/tests/test_shuffle.py
@@ -650,6 +650,26 @@ def test_set_index_timezone():
         d2.divisions[0] == s2badtype[0]
 
 
+@pytest.mark.parametrize("unit", ["ns", "us"])
+def test_set_index_datetime_precision(unit):
+    # https://github.com/dask/dask/issues/6864
+
+    df = pd.DataFrame(
+        [
+            [1567703791155681, 1],
+            [1567703792155681, 2],
+            [1567703790155681, 0],
+            [1567703793155681, 3],
+        ],
+        columns=["ts", "rank"],
+    )
+    df.ts = pd.to_datetime(df.ts, unit=unit)
+    ddf = dd.from_pandas(df, npartitions=2)
+    ddf = ddf.set_index("ts")
+
+    assert_eq(ddf, df.set_index("ts"))
+
+
 @pytest.mark.parametrize("drop", [True, False])
 def test_set_index_drop(drop):
     pdf = pd.DataFrame(