Repository: dask/dask

Issue:
Standardizing type for `divisions`
<!-- Please do a quick search of existing issues to make sure that this has not been asked before. -->
Looking through Dask's codebase, it seems like there isn't a consistent typing for a Dask object's `divisions`; in some places (like [`set_sorted_index`](https://github.com/dask/dask/blob/f5881891505b9a2ba2da195befb11ad7b4c7bb23/dask/dataframe/shuffle.py#L1017)), we return an object with a tuple `divisions`, while in others (such as [`set_partition`](https://github.com/dask/dask/blob/f5881891505b9a2ba2da195befb11ad7b4c7bb23/dask/dataframe/shuffle.py#L325)) we return an object with a list `divisions`. This becomes an issue in cases where we compare `divisions` between different objects, as we can run into cases where the elements contained in two objects' `divisions` are identical, but they are not seen as equal.

Some questions that come to mind:

- Is there an ideal type for `divisions`? I would assume tuples since `divisions` is generally treated as immutable even in the list case, but list functionality is used in several places in the codebase to assemble `divisions`.
- If there is an ideal type for `divisions`, how can we enforce it? It seems like one reason this problem exists is because in most places, list and tuple `divisions` function exactly the same - it is typically only when they are compared that issues arise. One potential solution would be to make `divisions` a property with a setter method that either:
    - Implicitly sets the input value to whatever type we desire `divisions` to be
    - Raises an error if the input value is not the proper `divisions` type
- If there is no ideal type for `divisions`, is there a workaround for comparisons?

cc @jsignell as I notice you are doing some work on `divisions` in #8379

EDIT:

To give additional context, I encountered this issue while debugging some breakage in dask-sql:

In some cases, when performing `JOIN` operations, dask-sql implicitly calls [`single_partition_join`](https://github.com/dask/dask/blob/f5881891505b9a2ba2da195befb11ad7b4c7bb23/dask/dataframe/multi.py#L389-L444) through `dd.merge`. Recently, #8341 did some refactoring to this function which, among other things, changed the divisions of the merged result from a tuple to a list (I don't think this was an intention of the PR, just a side effect).

This causes breakage later on in dask-sql if we attempt to subscript the result of this merge operation with a Series (something like `df[df[col].where(...)]`) with identical tuple divisions, as `DataFrame.__getitem__` does a divisions check to decide whether or not to `_maybe_align_partitions`:

https://github.com/dask/dask/blob/f5881891505b9a2ba2da195befb11ad7b4c7bb23/dask/dataframe/core.py#L4108-L4113

And `_maybe_align_partitions` does a divisions equality check to decide whether or not to actually `align_partitions` (which fails if not all `divisions` are known):

https://github.com/dask/dask/blob/f5881891505b9a2ba2da195befb11ad7b4c7bb23/dask/dataframe/multi.py#L166-L169

Here's a minimal reproducer of that particular issue:

```python
import dask.dataframe as dd
import pandas as pd

df = pd.DataFrame({"a": list(range(40))})
ddf = dd.from_pandas(df, npartitions=4)

cond = ddf.a > 20

# set unknown but inequal divisions
ddf.divisions = [None] * 5
cond.divisions = (None,) * 5

ddf[cond]
```


Discussion:
Thanks for raising this @charlesbluca !

At the moment, I think divisions *should* be a tuple.  However, I know there are many cases (both internally and down-stream) where lists are used. I also think it is worth mentioning here that I have been planning to write up a design proposal to support a dataframe-based divisions definitions to cover most of the motivating cases for multi-index support (without needing to support multi-index).  Given this background, I do like the idea of making divisions a property with a setter method. I don't think we need to raise an error when a user (or down-stream library) tries to set the divisions to a list, but we should automatically convert the type to tuple for now (and possibly support frame-based representations in the future).
> make `divisions` a property with a setter method that Implicitly sets the input value to whatever type we desire `divisions` to be

+1. For how critical it is, I've always been surprised that it gets no validation. This would probably fix most of the problems?

It could also be reasonable for `assert_eq`and `assert_divisions` to check the type of divisions. This would have caught https://github.com/dask/dask/pull/8389, for example.
I love the idea of adding this to `assert_divisions` and adding a setter. 
A side benefit of making divisions a property is that we can give it a docstring and incorporate a bunch of the new language from ``set_index``.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -188,7 +188,7 @@ def __dir__(self):
     @property
     def divisions(self):
         """Dummy divisions to be compat with Series and DataFrame"""
-        return [None, None]
+        return (None, None)
 
     def __repr__(self):
         name = self._name if len(self._name) < 10 else self._name[:7] + "..."
@@ -320,7 +320,7 @@ def __init__(self, dsk, name, meta, divisions):
                 f"{typename(type(meta))}"
             )
         self._meta = meta
-        self.divisions = tuple(divisions)
+        self._divisions = tuple(divisions)
 
     def __dask_graph__(self):
         return self.dask
@@ -355,6 +355,47 @@ def _rebuild(self, dsk, *, rename=None):
     def _constructor(self):
         return new_dd_object
 
+    @property
+    def divisions(self):
+        """
+        Tuple of ``npartitions + 1`` values, in ascending order, marking the
+        lower/upper bounds of each partition's index. Divisions allow Dask
+        to know which partition will contain a given value, significantly
+        speeding up operations like `loc`, `merge`, and `groupby` by not
+        having to search the full dataset.
+
+        Example: for ``divisions = (0, 10, 50, 100)``, there are three partitions,
+        where the index in each partition contains values [0, 10), [10, 50),
+        and [50, 100], respectively. Dask therefore knows ``df.loc[45]``
+        will be in the second partition.
+
+        When every item in ``divisions`` is ``None``, the divisions are unknown.
+        Most operations can still be performed, but some will be much slower,
+        and a few may fail.
+
+        It is uncommon to set ``divisions`` directly. Instead, use ``set_index``,
+        which sorts and splits the data as needed.
+        See https://docs.dask.org/en/latest/dataframe-design.html#partitions.
+        """
+        return self._divisions
+
+    @divisions.setter
+    def divisions(self, value):
+        if None in value:
+            if any(v is not None for v in value):
+                warnings.warn(
+                    "recieved `divisions` with mix of nulls and non-nulls, future versions will only accept "
+                    "`divisions` that are all null or all non-null",
+                    PendingDeprecationWarning,
+                )
+        if not isinstance(value, tuple):
+            warnings.warn(
+                f"recieved `divisions` of type {type(value)}, future versions will only accept `divisions` of type "
+                "tuple",
+                PendingDeprecationWarning,
+            )
+        self._divisions = value
+
     @property
     def npartitions(self):
         """Return number of partitions"""
@@ -395,7 +436,7 @@ def __getstate__(self):
         return self._args
 
     def __setstate__(self, state):
-        self.dask, self._name, self._meta, self.divisions = state
+        self.dask, self._name, self._meta, self._divisions = state
 
     def copy(self, deep=False):
         """Make a copy of the dataframe
diff --git a/dask/dataframe/shuffle.py b/dask/dataframe/shuffle.py
--- a/dask/dataframe/shuffle.py
+++ b/dask/dataframe/shuffle.py
@@ -320,7 +320,7 @@ def set_partition(
             column_dtype=df.columns.dtype,
         )
 
-    df4.divisions = methods.tolist(divisions)
+    df4.divisions = tuple(methods.tolist(divisions))
 
     return df4.map_partitions(M.sort_index)
 
diff --git a/dask/dataframe/utils.py b/dask/dataframe/utils.py
--- a/dask/dataframe/utils.py
+++ b/dask/dataframe/utils.py
@@ -588,6 +588,9 @@ def assert_dask_graph(dask, label):
 def assert_divisions(ddf):
     if not hasattr(ddf, "divisions"):
         return
+
+    assert isinstance(ddf.divisions, tuple)
+
     if not getattr(ddf, "known_divisions", False):
         return
 


Test patch:
diff --git a/dask/dataframe/tests/test_indexing.py b/dask/dataframe/tests/test_indexing.py
--- a/dask/dataframe/tests/test_indexing.py
+++ b/dask/dataframe/tests/test_indexing.py
@@ -335,7 +335,7 @@ def test_loc_on_numpy_datetimes():
         {"x": [1, 2, 3]}, index=list(map(np.datetime64, ["2014", "2015", "2016"]))
     )
     a = dd.from_pandas(df, 2)
-    a.divisions = list(map(np.datetime64, a.divisions))
+    a.divisions = tuple(map(np.datetime64, a.divisions))
 
     assert_eq(a.loc["2014":"2015"], a.loc["2014":"2015"])
 
@@ -345,7 +345,7 @@ def test_loc_on_pandas_datetimes():
         {"x": [1, 2, 3]}, index=list(map(pd.Timestamp, ["2014", "2015", "2016"]))
     )
     a = dd.from_pandas(df, 2)
-    a.divisions = list(map(pd.Timestamp, a.divisions))
+    a.divisions = tuple(map(pd.Timestamp, a.divisions))
 
     assert_eq(a.loc["2014":"2015"], a.loc["2014":"2015"])
 
diff --git a/dask/dataframe/tests/test_merge_column_and_index.py b/dask/dataframe/tests/test_merge_column_and_index.py
--- a/dask/dataframe/tests/test_merge_column_and_index.py
+++ b/dask/dataframe/tests/test_merge_column_and_index.py
@@ -3,7 +3,6 @@
 import pytest
 
 import dask.dataframe as dd
-from dask.dataframe.multi import single_partition_join
 from dask.dataframe.utils import assert_eq
 
 
@@ -245,23 +244,3 @@ def test_merge_column_with_nulls(repartition):
     dask_result = df1_d.merge(df2_d, how="left", left_on="a", right_index=True)
 
     assert_eq(dask_result, pandas_result)
-
-
-@pytest.mark.parametrize("how", ["inner", "left", "right"])
-def test_single_column_join_regression(
-    ddf_left_single, ddf_right_single, on, how, shuffle_method
-):
-    # won't need this if we standardize divisions typing
-    # see https://github.com/dask/dask/issues/8388
-
-    # compute expected
-    expected = ddf_left_single.merge(
-        ddf_right_single, on=on, how=how, shuffle=shuffle_method
-    )
-
-    # compute with internal function
-    result = single_partition_join(ddf_left_single, ddf_right_single, on=on, how=how)
-
-    assert_eq(result, expected)
-    assert result.divisions == expected.divisions
-    assert len(result.__dask_graph__()) < 30
diff --git a/dask/dataframe/tests/test_multi.py b/dask/dataframe/tests/test_multi.py
--- a/dask/dataframe/tests/test_multi.py
+++ b/dask/dataframe/tests/test_multi.py
@@ -335,7 +335,7 @@ def test_merge_asof_left_on_right_index(
     b = dd.from_pandas(B, npartitions=2)
 
     if unknown_divisions:
-        a.divisions = [None] * len(a.divisions)
+        a.divisions = (None,) * len(a.divisions)
 
     C = pd.merge_asof(
         A,
@@ -368,7 +368,7 @@ def test_merge_asof_left_on_right_index(
             b = dd.from_pandas(B, npartitions=nparts)
 
             if unknown_divisions:
-                a.divisions = [None] * len(a.divisions)
+                a.divisions = (None,) * len(a.divisions)
 
             C = pd.merge_asof(
                 A,