Repository: dask/dask

Issue:
DataFrame merge: clarify that merging on column puts all matching values into one partition
I naively tried to do `dd.merge(a, b, on="column_with_ten_values")`, where `a` and `b` were both large DataFrames with thousands of partitions each.

Eventually the compute failed with:
```python-traceback
[File /opt/conda/envs/coiled/lib/python3.9/site-packages/dask/dataframe/multi.py:275, in merge_chunk()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/frame.py:9329, in merge()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:122, in merge()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:716, in get_result()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:967, in _get_join_info()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:941, in _get_join_indexers()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:1509, in get_join_indexers()

File ~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx:48, in pandas._libs.join.inner_join()
     ]()[46](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=45)[             count += lc * rc
     ]()[47](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=46)[ 
---> ]()[48](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=47)[ left_indexer = np.empty(count, dtype=np.intp)
     ]()[49](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=48)[ right_indexer = np.empty(count, dtype=np.intp)
     ]()[50](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=49)[ 

MemoryError: Unable to allocate 224. TiB for an array with shape (30824647463820,) and data type int64]()
```

I was confused why a 224 TiB array needed to be allocated. As a user, this was a strange and concerning error to see. Eventually I realized that my `on` column had only a few unique values, so (I think) it was trying to concatenate all those values across all partitions into one during the shuffle stage of the hash join. Because the shuffle ensures that:

> After this operation all elements that have the same index will be in the same partition.

It would be good to:
1. Document clearly that when merging on a column, every row with the same value needs to be able to fit into one partition. If there are only a few unique values for the column, but many rows, you'll likely encounter an error.
2. Catch/preemptively check for this error at runtime, and reraise a more informative error to users. Not sure what alternative we'd suggest though. The fact that divisions, and by extension shuffling, can't handle repeated values (xref https://github.com/dask/dask/issues/8437) I think makes this fundamentally unsupported right now?

cc @jsignell 


Discussion:
The documentation task seems like a good-first-issue :)

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -5108,6 +5108,10 @@ def merge(
         3. Joining both on columns. In this case a hash join is performed using
            ``dask.dataframe.multi.hash_join``.
 
+        In some cases, you may see a ``MemoryError`` if the ``merge`` operation requires
+        an internal ``shuffle``, because shuffling places all rows that have the same
+        index in the same partition. To avoid this error, make sure all rows with the
+        same ``on``-column value can fit on a single partition.
         """
 
         if not is_dataframe_like(right):


Test patch: