Repository: dask/dask

Issue:
Setting divisions on dask dataframe collapses rows. 
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**: Trying to set divisions on a dask dataframe created from delayed objects results in rows being erased from the dataframe.

**What you expected to happen**: The divisions to work as expected

**Minimal Complete Verifiable Example**:

```python
import pandas as pd
import dask.dataframe as dd
def get_df(bid):
    df = pd.DataFrame({"id": [bid, bid, bid], "value": [bid+0.1, bid+0.2, bid+0.3]})
    df = df.set_index('id')
    return df

ddf = dd.from_delayed([dask.delayed(get_df)(i) for i in range(5)])
ddf.divisions=[0, 9, 15]
ddf.compute()
```
The code outputs this:

![image](https://user-images.githubusercontent.com/12487392/158036601-09fbbe67-d370-44fc-a106-5d490e22f181.png)

Somehow, rest of the rows is lost. The output should have been like this:
![image](https://user-images.githubusercontent.com/12487392/158036643-bc529baf-956c-46c2-85f8-862df8975ec3.png)

**Anything else we need to know?**:
The two partitions of ddf contains the following:
![image](https://user-images.githubusercontent.com/12487392/158036621-7db5df77-12c1-48c1-a89b-7961e96ebe6f.png)
![image](https://user-images.githubusercontent.com/12487392/158036628-7fe31217-5673-4972-886a-962e28c6608d.png)

**Environment**: MAC

- Dask version: 2022.2.1
- Python version: 3.9.7
- Operating System: Mac Catalina
- Install method (conda, pip, source): conda

<!-- If you are reporting an issue such as scale stability, cluster deadlock.
Please provide a cluster dump state with this issue, by running client.dump_cluster_state()

https://distributed.dask.org/en/stable/api.html?highlight=dump_cluster_state#distributed.Client.dump_cluster_state

-->

<details>
<summary>Cluster Dump State:</summary>

</details>


Discussion:
Assigning directly to divisions (`ddf.divisions = ...`) has no error checking and isn't really recommended. In your case the issue is that the divisions you're assigning only contain 2 partitions, while your original dataframe has 5 partitions. This results in the trailing 3 partitions being implicitly dropped. As a reminder, divisions is a tuple of npartitions + 1 elements, where the first npartitions elements are the minimum of their partition, and the final value is the maximum value in the last partition.

If you know the divisions of your output dataframe, I recommend passing them in to `dd.from_delayed` directly via the `divisions` kwarg, not mutating the value on the dataframe directly.
The only action item I see on our end here would be adding a nice error message when users assign divisions directly and the divisions don't match the current number of partitions. This would hopefully help with users debugging cases like this.
@jcrist Thanks for the response.

I think there is more going on. I have not been able to create two partitions with (0, 9, 15) divisions from the list of 5 delayed objects, no matter which approach I use.

Consider the following:

```python
import pandas as pd
import dask.dataframe as dd
def get_df(bid):
    df = pd.DataFrame({"id": [bid, bid, bid], "value": [bid+0.1, bid+0.2, bid+0.3]})
    df = df.set_index('id')
    return df

ddf = dd.from_delayed([dask.delayed(get_df)(i) for i in range(5)], divisions=(0,3,6,9,12,15))
# ddf = ddf.repartition(npartitions=2)  # adding this line won't help either
ddf = ddf.repartition(divisions=(0, 9, 15))
ddf.compute()
```
This still results in missing rows.

Basically, how do you correctly repartition the ddf created from the 5 delayed objects into two partition with (0, 9, 15) divisions? 

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -323,7 +323,7 @@ def __init__(self, dsk, name, meta, divisions):
                 f"{typename(type(meta))}"
             )
         self._meta = meta
-        self._divisions = tuple(divisions)
+        self.divisions = tuple(divisions)
 
     def __dask_graph__(self):
         return self.dask
@@ -384,19 +384,31 @@ def divisions(self):
 
     @divisions.setter
     def divisions(self, value):
+        if not isinstance(value, tuple):
+            raise TypeError("divisions must be a tuple")
+
+        if hasattr(self, "_divisions") and len(value) != len(self._divisions):
+            n = len(self._divisions)
+            raise ValueError(
+                f"This dataframe has npartitions={n - 1}, divisions should be a "
+                f"tuple of length={n}, got {len(value)}"
+            )
+
         if None in value:
             if any(v is not None for v in value):
-                warnings.warn(
-                    "recieved `divisions` with mix of nulls and non-nulls, future versions will only accept "
-                    "`divisions` that are all null or all non-null",
-                    PendingDeprecationWarning,
+                raise ValueError(
+                    "divisions may not contain a mix of None and non-None values"
                 )
-        if not isinstance(value, tuple):
-            warnings.warn(
-                f"recieved `divisions` of type {type(value)}, future versions will only accept `divisions` of type "
-                "tuple",
-                PendingDeprecationWarning,
-            )
+        else:
+            # Known divisions, check monotonically increasing
+
+            # XXX: if the index dtype is an ordered categorical dtype, then we skip the
+            # sortedness check, since the order is dtype dependent
+            index_dtype = getattr(self._meta, "index", self._meta).dtype
+            if not (is_categorical_dtype(index_dtype) and index_dtype.ordered):
+                if value != tuple(sorted(value)):
+                    raise ValueError("divisions must be sorted")
+
         self._divisions = value
 
     @property
@@ -1318,8 +1330,10 @@ def repartition(
             For convenience if given an integer this will defer to npartitions
             and if given a string it will defer to partition_size (see below)
         npartitions : int, optional
-            Number of partitions of output. Only used if partition_size
-            isn't specified.
+            Approximate number of partitions of output. Only used if partition_size
+            isn't specified. The number of partitions used may be slightly
+            lower than npartitions depending on data distribution, but will never be
+            higher.
         partition_size: int or string, optional
             Max number of bytes of memory for each partition. Use numbers or
             strings like 5MB. If specified npartitions and divisions will be
@@ -3495,13 +3509,13 @@ def rename(self, index=None, inplace=False, sorted_index=False):
                             "isn't monotonic_increasing"
                         )
                         raise ValueError(msg)
-                    res.divisions = tuple(methods.tolist(new))
+                    res._divisions = tuple(methods.tolist(new))
                 else:
                     res = res.clear_divisions()
             if inplace:
                 self.dask = res.dask
                 self._name = res._name
-                self.divisions = res.divisions
+                self._divisions = res.divisions
                 self._meta = res._meta
                 res = self
         return res
@@ -4051,7 +4065,7 @@ class Index(Series):
     }
 
     def __getattr__(self, key):
-        if is_categorical_dtype(self.dtype) and key in self._cat_attributes:
+        if is_categorical_dtype(self._meta.dtype) and key in self._cat_attributes:
             return getattr(self.cat, key)
         elif key in self._dt_attributes:
             return getattr(self.dt, key)
@@ -4404,7 +4418,7 @@ def __setitem__(self, key, value):
         self.dask = df.dask
         self._name = df._name
         self._meta = df._meta
-        self.divisions = df.divisions
+        self._divisions = df.divisions
 
     def __delitem__(self, key):
         result = self.drop([key], axis=1)
@@ -5869,7 +5883,7 @@ def handle_out(out, result):
         out.dask = result.dask
 
         if not isinstance(out, Scalar):
-            out.divisions = result.divisions
+            out._divisions = result.divisions
     elif out is not None:
         msg = (
             "The out parameter is not fully supported."
@@ -6275,7 +6289,13 @@ def map_partitions(
         if collections:
             simple = False
 
-    divisions = dfs[0].divisions
+    if align_dataframes:
+        divisions = dfs[0].divisions
+    else:
+        # Unaligned, dfs is a mix of 1 partition and 1+ partition dataframes,
+        # use longest divisions found
+        divisions = max((d.divisions for d in dfs), key=len)
+
     if transform_divisions and isinstance(dfs[0], Index) and len(dfs) == 1:
         try:
             divisions = func(
@@ -6612,7 +6632,7 @@ def cov_corr(df, min_periods=None, corr=False, scalar=False, split_every=False):
     meta = make_meta(
         [(c, "f8") for c in df.columns], index=df.columns, parent_meta=df._meta
     )
-    return DataFrame(graph, name, meta, (df.columns[0], df.columns[-1]))
+    return DataFrame(graph, name, meta, (df.columns.min(), df.columns.max()))
 
 
 def cov_corr_chunk(df, corr=False):
@@ -7078,6 +7098,8 @@ def repartition_npartitions(df, npartitions):
             divisions[0] = df.divisions[0]
             divisions[-1] = df.divisions[-1]
 
+            # Ensure the computed divisions are unique
+            divisions = list(unique(divisions[:-1])) + [divisions[-1]]
             return df.repartition(divisions=divisions)
         else:
             div, mod = divmod(npartitions, df.npartitions)
diff --git a/dask/dataframe/indexing.py b/dask/dataframe/indexing.py
--- a/dask/dataframe/indexing.py
+++ b/dask/dataframe/indexing.py
@@ -231,11 +231,19 @@ def _loc_slice(self, iindexer, cindexer):
             stop = self.obj.npartitions - 1
 
         if iindexer.start is None and self.obj.known_divisions:
-            istart = self.obj.divisions[0]
+            istart = (
+                self.obj.divisions[0]
+                if iindexer.stop is None
+                else min(self.obj.divisions[0], iindexer.stop)
+            )
         else:
             istart = self._coerce_loc_index(iindexer.start)
         if iindexer.stop is None and self.obj.known_divisions:
-            istop = self.obj.divisions[-1]
+            istop = (
+                self.obj.divisions[-1]
+                if iindexer.start is None
+                else max(self.obj.divisions[-1], iindexer.start)
+            )
         else:
             istop = self._coerce_loc_index(iindexer.stop)
 
diff --git a/dask/dataframe/shuffle.py b/dask/dataframe/shuffle.py
--- a/dask/dataframe/shuffle.py
+++ b/dask/dataframe/shuffle.py
@@ -15,6 +15,7 @@
 from dask.dataframe import methods
 from dask.dataframe.core import DataFrame, Series, _Frame, map_partitions, new_dd_object
 from dask.dataframe.dispatch import group_split_dispatch, hash_object_dispatch
+from dask.dataframe.utils import UNKNOWN_CATEGORIES
 from dask.highlevelgraph import HighLevelGraph
 from dask.layers import ShuffleLayer, SimpleShuffleLayer
 from dask.sizeof import sizeof
@@ -60,6 +61,9 @@ def _calculate_divisions(
         except (TypeError, ValueError):  # str type
             indexes = np.linspace(0, n - 1, npartitions + 1).astype(int)
             divisions = [divisions[i] for i in indexes]
+    else:
+        # Drop duplicate divisions returned by partition quantiles
+        divisions = list(toolz.unique(divisions[:-1])) + [divisions[-1]]
 
     mins = remove_nans(mins)
     maxes = remove_nans(maxes)
@@ -308,6 +312,12 @@ def set_partition(
     if pd.isna(divisions).any() and pd.api.types.is_integer_dtype(dtype):
         # Can't construct a Series[int64] when any / all of the divisions are NaN.
         divisions = df._meta._constructor_sliced(divisions)
+    elif (
+        pd.api.types.is_categorical_dtype(dtype)
+        and UNKNOWN_CATEGORIES in dtype.categories
+    ):
+        # If categories are unknown, leave as a string dtype instead.
+        divisions = df._meta._constructor_sliced(divisions)
     else:
         divisions = df._meta._constructor_sliced(divisions, dtype=dtype)
 
@@ -1055,7 +1065,7 @@ def compute_divisions(df, col=None, **kwargs) -> tuple:
 def compute_and_set_divisions(df, **kwargs):
     mins, maxes, lens = _compute_partition_stats(df.index, allow_overlap=True, **kwargs)
     if len(mins) == len(df.divisions) - 1:
-        df.divisions = tuple(mins) + (maxes[-1],)
+        df._divisions = tuple(mins) + (maxes[-1],)
         if not any(mins[i] >= maxes[i - 1] for i in range(1, len(mins))):
             return df
 


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -1078,15 +1078,19 @@ def test_align_dataframes():
     df1 = pd.DataFrame({"A": [1, 2, 3, 3, 2, 3], "B": [1, 2, 3, 4, 5, 6]})
     df2 = pd.DataFrame({"A": [3, 1, 2], "C": [1, 2, 3]})
 
-    def merge(a, b):
-        res = pd.merge(a, b, left_on="A", right_on="A", how="left")
-        return res
-
-    expected = merge(df1, df2)
-
     ddf1 = dd.from_pandas(df1, npartitions=2)
-    actual = ddf1.map_partitions(merge, df2, align_dataframes=False)
+    ddf2 = dd.from_pandas(df2, npartitions=1)
+
+    actual = ddf1.map_partitions(
+        pd.merge, df2, align_dataframes=False, left_on="A", right_on="A", how="left"
+    )
+    expected = pd.merge(df1, df2, left_on="A", right_on="A", how="left")
+    assert_eq(actual, expected, check_index=False, check_divisions=False)
 
+    actual = ddf2.map_partitions(
+        pd.merge, ddf1, align_dataframes=False, left_on="A", right_on="A", how="right"
+    )
+    expected = pd.merge(df2, df1, left_on="A", right_on="A", how="right")
     assert_eq(actual, expected, check_index=False, check_divisions=False)
 
 
diff --git a/dask/dataframe/tests/test_indexing.py b/dask/dataframe/tests/test_indexing.py
--- a/dask/dataframe/tests/test_indexing.py
+++ b/dask/dataframe/tests/test_indexing.py
@@ -45,6 +45,8 @@ def test_loc():
 
     pytest.raises(KeyError, lambda: d.loc[1000])
     assert_eq(d.loc[1000:], full.loc[1000:])
+    assert_eq(d.loc[1000:2000], full.loc[1000:2000])
+    assert_eq(d.loc[:-1000], full.loc[:-1000])
     assert_eq(d.loc[-2000:-1000], full.loc[-2000:-1000])
 
     assert sorted(d.loc[5].dask) == sorted(d.loc[5].dask)
diff --git a/dask/dataframe/tests/test_multi.py b/dask/dataframe/tests/test_multi.py
--- a/dask/dataframe/tests/test_multi.py
+++ b/dask/dataframe/tests/test_multi.py
@@ -2193,7 +2193,7 @@ def test_multi_duplicate_divisions():
 
     ddf1 = dd.from_pandas(df1, npartitions=2).set_index("x")
     ddf2 = dd.from_pandas(df2, npartitions=1).set_index("x")
-    assert ddf1.npartitions == 2
+    assert ddf1.npartitions <= 2
     assert len(ddf1) == len(df1)
 
     r1 = ddf1.merge(ddf2, how="left", left_index=True, right_index=True)
diff --git a/dask/dataframe/tests/test_shuffle.py b/dask/dataframe/tests/test_shuffle.py
--- a/dask/dataframe/tests/test_shuffle.py
+++ b/dask/dataframe/tests/test_shuffle.py
@@ -663,7 +663,7 @@ def test_set_index_interpolate(engine):
         d = dd.from_pandas(df, 2)
 
     d1 = d.set_index("x", npartitions=3)
-    assert d1.npartitions == 3
+    assert d1.npartitions <= 3
     assert set(d1.divisions) == {1, 2, 4}
 
     d2 = d.set_index("y", npartitions=3)