Repository: dask/dask

Issue:
Joins return incorrect results if performed after index map
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**: Joining dask objects return incorrect results if the join operation was performed after the index of the dask object was modified by a map operation beforehand.  Resetting then setting the index again fixes this issue, but documentation does not seem to mention if this behaviour is intended or not

**What you expected to happen**: The join should work as if it was performed in pandas, taking into account the new index values.

**Minimal Complete Verifiable Example**:

```python
import dask
import numpy as np
import pandas as pd

original = dask.datasets.timeseries()
another_index = pd.date_range(
    start=np.datetime64('1999-12-31 00:00:00'), end=np.datetime64('2000-01-29 23:59:59'), freq='1s')
print(len(original.index.compute().join(another_index, how='inner')))

original.index = original.index.map(
    lambda x: x - np.timedelta64(1, 'D'), meta=('timestamp', 'datetime64[ns]'))
another_dataframe = pd.DataFrame(index=another_index)
print(len(original.index.compute().join(another_index, how='inner')))

first_join = original.join(another_dataframe, how='inner')
print(len(first_join.index.compute()))

original = original.reset_index()
original = original.set_index('timestamp', npartitions='auto')
second_join = original.join(another_dataframe, how='inner')
print(len(second_join.index.compute()))
```

Output :
2505600 --- Old and new index have overlapping elements
2592000 --- After mapping indexes are equivalent
0 --- First dask join fails, does not take into account the old or new index
2592000 --- Correct result only after resetting and setting the index again

**Environment**:

- Dask version: 2.15.0
- Python version: 3.6.9
- Operating System: Ubuntu 18.04.4 LTS
- Install method (conda, pip, source): pip



Discussion:
Hi @velezsarain thanks for opening this issue! This is a real issue although it's not clear to me how to address it. I guess we need to be sure to do the index assignment before we do any joins. I have been looking at the task graphs (`.visualize()`) of `original` and `first_join` to try to get some insight into what is going on. I will be able to take a deeper look tomorrow.
@velezsarain are your index partitions valid after assigning the index?

If I clear the divisions then the numbers seem correct

```
In [24]: original = original.clear_divisions()

In [25]: another_dataframe = pd.DataFrame(index=another_index)
    ...: print(len(original.index.compute().join(another_index, how='inner')))
    ...:
    ...: first_join = original.join(another_dataframe, how='inner')
    ...: print(len(first_join.index.compute()))
    ...:
    ...: original = original.reset_index()
    ...: original = original.set_index('timestamp', npartitions='auto')
    ...: second_join = original.join(another_dataframe, how='inner')
    ...: print(len(second_join.index.compute()))
2592000
2592000
2592000
```

I'd *strongly* recommend against setting the index like that. Better to use `set_index`, which I think would have handled this for you.

```python
In [30]: original = original.set_index(original.index.map(
    ...:     lambda x: x - np.timedelta64(1, 'D'), meta=('timestamp', 'datetime64[ns]')))
    ...:
    ...:

In [31]: original.divisions
Out[31]:
(Timestamp('1999-12-31 00:00:00', freq='S'),
 Timestamp('2000-01-01 00:00:00', freq='S'),
 Timestamp('2000-01-02 00:00:00', freq='S'),
...
```

It *might* be worth overriding `_Frame.index.setter` to not allow setting via `.`. Do you have thoughts on that @jsignell?
Looking briefly at `index.setter`

```python
    @index.setter
    def index(self, value):
        self.divisions = value.divisions
        result = map_partitions(
            methods.assign_index, self, value, enforce_metadata=False
        )
        self.dask = result.dask
        self._name = result._name
        self._meta = result._meta
```

the issue points to the *real* problem, that the divisions in `index.map(...)` are incorrect. If they were correct, then things would be fine. In `map_partitions` we have a warning

```
        Note that the index and divisions are assumed to remain unchanged.
```

We might consider overriding the docstring for `Index.map` to include a similar warning (right now Index.map is just Series.map).
@TomAugspurger You are correct, I assumed that this issue wasn't a bug, but I found nothing when I searched the documentation for the index. In pandas setting the index in this manner does not lead to errors, inducing my mistake.

The code I'm working on only has 3 dropped elements after the join out of  several thousand, these elements correspond to the items that were left outside of the original divisions.
I think the best short-term option is to define `Index.map` with a docstring that contains a similar warning to `.map_partitions`, that the divisions are assumed to be unchanged, and should be cleared or set manually if they are (or, I guess another option is to apply the mapping to the scalar division values? I'm not sure how I feel about that.)
Thanks @TomAugspurger! I would be fine with not allowing dot assignment for indexes if it is not recommended and can lead to bad results. I'd rather throw an error and tell people to use `set_index` instead. I think a warning in `index.map` would also be a good addition.
Unfortunately while fixing this issue on my project I ran into another issue with the join operation. I'm currently trying to reproduce to see if it is related to index mapping but the gist of it is that an inner join is duplicating one of my dataframes, returning me more items that I had to begin with.
I've managed to reproduce, but I figured out it is not related to index mapping, so I will be opening another issue for this bug.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -3015,7 +3015,7 @@ def map(self, arg, na_action=None, meta=no_default):
         else:
             meta = make_meta(meta, index=getattr(make_meta(self), "index", None))
 
-        return Series(graph, name, meta, self.divisions)
+        return type(self)(graph, name, meta, self.divisions)
 
     @derived_from(pd.Series)
     def dropna(self):
@@ -3383,6 +3383,26 @@ def to_frame(self, index=True, name=None):
             else:
                 return self.map_partitions(M.to_frame, meta=self._meta.to_frame())
 
+    @insert_meta_param_description(pad=12)
+    @derived_from(pd.Index)
+    def map(self, arg, na_action=None, meta=no_default, is_monotonic=False):
+        """
+        Note that this method clears any known divisions.
+
+        If your mapping function is monotonically increasing then use `is_monotonic`
+        to apply the maping function to the old divisions and assign the new
+        divisions to the output.
+
+        """
+        applied = super(Index, self).map(arg, na_action=na_action, meta=meta)
+        if is_monotonic and self.known_divisions:
+            applied.divisions = tuple(
+                pd.Series(self.divisions).map(arg, na_action=na_action)
+            )
+        else:
+            applied = applied.clear_divisions()
+        return applied
+
 
 class DataFrame(_Frame):
     """


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -4103,6 +4103,19 @@ def test_meta_error_message():
     assert "pandas" in str(info.value)
 
 
+def test_map_index():
+    df = pd.DataFrame({"x": [1, 2, 3, 4, 5]})
+    ddf = dd.from_pandas(df, npartitions=2)
+    assert ddf.known_divisions is True
+
+    cleared = ddf.index.map(lambda x: x * 10)
+    assert cleared.known_divisions is False
+
+    applied = ddf.index.map(lambda x: x * 10, is_monotonic=True)
+    assert applied.known_divisions is True
+    assert applied.divisions == tuple(x * 10 for x in ddf.divisions)
+
+
 def test_assign_index():
     df = pd.DataFrame({"x": [1, 2, 3, 4, 5]})
     ddf = dd.from_pandas(df, npartitions=2)