Repository: dask/dask

Issue:
IndexError while merging dask dataframes
I'm running into the following issue while merging a dask dataframe with an empty dask dataframe.
**IndexError: single positional indexer is out-of-bounds**

```python-traceback
The stacktrace is as follows: 
File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/multi.py", line 555, in merge
    indicator=indicator,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/multi.py", line 271, in hash_join
    lhs2 = shuffle_func(lhs, left_on, npartitions=npartitions, shuffle=shuffle)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/shuffle.py", line 239, in shuffle
    meta=series_type_like_df(df)([0]),
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/utils.py", line 674, in series_type_like_df
    return type(df._meta.iloc[:, 0])
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 1418, in __getitem__
    return self._getitem_tuple(key)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 2092, in _getitem_tuple
    self._has_valid_tuple(tup)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 235, in _has_valid_tuple
    self._validate_key(k, i)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 2014, in _validate_key
    self._validate_integer(key, axis)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 2088, in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds
```

This error is specifically happening with the latest release (dask==2.5.2) . It seems to work smoothly with dask 2.4.0
IndexError while merging dask dataframes
I'm running into the following issue while merging a dask dataframe with an empty dask dataframe.
**IndexError: single positional indexer is out-of-bounds**

```python-traceback
The stacktrace is as follows: 
File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/multi.py", line 555, in merge
    indicator=indicator,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/multi.py", line 271, in hash_join
    lhs2 = shuffle_func(lhs, left_on, npartitions=npartitions, shuffle=shuffle)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/shuffle.py", line 239, in shuffle
    meta=series_type_like_df(df)([0]),
  File "/home/ubuntu/.local/lib/python3.6/site-packages/dask/dataframe/utils.py", line 674, in series_type_like_df
    return type(df._meta.iloc[:, 0])
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 1418, in __getitem__
    return self._getitem_tuple(key)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 2092, in _getitem_tuple
    self._has_valid_tuple(tup)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 235, in _has_valid_tuple
    self._validate_key(k, i)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 2014, in _validate_key
    self._validate_integer(key, axis)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/indexing.py", line 2088, in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds
```

This error is specifically happening with the latest release (dask==2.5.2) . It seems to work smoothly with dask 2.4.0


Discussion:
Thank you for the error report @debabrata-mishra .  Are you able to provide a minimal reproducible example?  That is probably the first step to identifying and resolving this error:

-  https://stackoverflow.com/help/minimal-reproducible-example
-  https://blog.dask.org/2018/02/28/minimal-bug-reports
Sure. The following is the code snippet where the code used to fail. 

		overall_data = pd.DataFrame()
		overall_data = dd.from_pandas(overall_df, npartitions=self.npartitions)

		for x in range(self.k_clusters):
			#some code blocks
			df = df[df.cluster == x]
			overall_data = dd.merge(overall_data, df, how='outer')

The following is a sample code which should be able to reproduce the error:

```
import pandas as pd
import dask.dataframe as dd
k_clusters = 3
df = pd.DataFrame({'user': ['A', 'B', 'C', 'D', 'E', 'F'],
                   'cluster': [1, 1, 2, 2, 3, 3]})
df = dd.from_pandas(df, npartitions=10)

empty_df = pd.DataFrame()
empty_df = dd.from_pandas(empty_df, npartitions=10)

for x in range(k_clusters):
	df = df[df.cluster == x]
	empty_df = dd.merge(empty_df, df, how='outer')
```
As mentioned the code snipet works flawlessly in dask v2.4.0 but is throwing the aforementioned error with the latest release.
Thank you for the minimal reproducible example @debabrata-mishra . This is very helpful.

@rjzamora I suspect that this might have something to do with your recent work.  Would you mind taking a look?
No problem - I will look into this.  I can confirm that I get the same error
Thank you for the error report @debabrata-mishra .  Are you able to provide a minimal reproducible example?  That is probably the first step to identifying and resolving this error:

-  https://stackoverflow.com/help/minimal-reproducible-example
-  https://blog.dask.org/2018/02/28/minimal-bug-reports
Sure. The following is the code snippet where the code used to fail. 

		overall_data = pd.DataFrame()
		overall_data = dd.from_pandas(overall_df, npartitions=self.npartitions)

		for x in range(self.k_clusters):
			#some code blocks
			df = df[df.cluster == x]
			overall_data = dd.merge(overall_data, df, how='outer')

The following is a sample code which should be able to reproduce the error:

```
import pandas as pd
import dask.dataframe as dd
k_clusters = 3
df = pd.DataFrame({'user': ['A', 'B', 'C', 'D', 'E', 'F'],
                   'cluster': [1, 1, 2, 2, 3, 3]})
df = dd.from_pandas(df, npartitions=10)

empty_df = pd.DataFrame()
empty_df = dd.from_pandas(empty_df, npartitions=10)

for x in range(k_clusters):
	df = df[df.cluster == x]
	empty_df = dd.merge(empty_df, df, how='outer')
```
As mentioned the code snipet works flawlessly in dask v2.4.0 but is throwing the aforementioned error with the latest release.
Thank you for the minimal reproducible example @debabrata-mishra . This is very helpful.

@rjzamora I suspect that this might have something to do with your recent work.  Would you mind taking a look?
No problem - I will look into this.  I can confirm that I get the same error

Resolving patch:
diff --git a/dask/dataframe/shuffle.py b/dask/dataframe/shuffle.py
--- a/dask/dataframe/shuffle.py
+++ b/dask/dataframe/shuffle.py
@@ -15,7 +15,7 @@
 from ..highlevelgraph import HighLevelGraph
 from ..sizeof import sizeof
 from ..utils import digit, insert, M
-from .utils import series_type_like_df, hash_object_dispatch, group_split_dispatch
+from .utils import hash_object_dispatch, group_split_dispatch
 
 
 def set_index(
@@ -170,8 +170,8 @@ def set_partition(
     shuffle
     partd
     """
-    divisions = series_type_like_df(df)(divisions)
-    meta = series_type_like_df(df)([0])
+    divisions = df._meta._constructor_sliced(divisions)
+    meta = df._meta._constructor_sliced([0])
     if np.isscalar(index):
         partitions = df[index].map_partitions(
             set_partitions_pre, divisions=divisions, meta=meta
@@ -235,7 +235,7 @@ def shuffle(df, index, shuffle=None, npartitions=None, max_branch=32, compute=No
     partitions = index.map_partitions(
         partitioning_index,
         npartitions=npartitions or df.npartitions,
-        meta=series_type_like_df(df)([0]),
+        meta=df._meta._constructor_sliced([0]),
         transform_divisions=False,
     )
     df2 = df.assign(_partitions=partitions)
@@ -254,8 +254,8 @@ def shuffle(df, index, shuffle=None, npartitions=None, max_branch=32, compute=No
 
 def rearrange_by_divisions(df, column, divisions, max_branch=None, shuffle=None):
     """ Shuffle dataframe so that column separates along divisions """
-    divisions = series_type_like_df(df)(divisions)
-    meta = series_type_like_df(df)([0])
+    divisions = df._meta._constructor_sliced(divisions)
+    meta = df._meta._constructor_sliced([0])
     # Assign target output partitions to every row
     partitions = df[column].map_partitions(
         set_partitions_pre, divisions=divisions, meta=meta
diff --git a/dask/dataframe/utils.py b/dask/dataframe/utils.py
--- a/dask/dataframe/utils.py
+++ b/dask/dataframe/utils.py
@@ -679,13 +679,6 @@ def index_summary(idx, name=None):
     return "{}: {} entries{}".format(name, n, summary)
 
 
-def series_type_like_df(df):
-    """Get the concrete series type compatible with df."""
-    # TODO: https://github.com/pandas-dev/pandas/issues/27824
-    # Use a standard API, which will handle empty DataFrames.
-    return type(df._meta.iloc[:, 0])
-
-
 ###############################################################
 # Testing
 ###############################################################


Test patch:
diff --git a/dask/dataframe/tests/test_multi.py b/dask/dataframe/tests/test_multi.py
--- a/dask/dataframe/tests/test_multi.py
+++ b/dask/dataframe/tests/test_multi.py
@@ -1963,3 +1963,20 @@ def test_multi_duplicate_divisions():
     r2 = sf1.merge(sf2, how="left", left_index=True, right_index=True)
 
     assert_eq(r1, r2)
+
+
+def test_merge_outer_empty():
+    # Issue #5470 bug reproducer
+    k_clusters = 3
+    df = pd.DataFrame(
+        {"user": ["A", "B", "C", "D", "E", "F"], "cluster": [1, 1, 2, 2, 3, 3]}
+    )
+    df = dd.from_pandas(df, npartitions=10)
+    empty_df = dd.from_pandas(pd.DataFrame(), npartitions=10)
+
+    for x in range(0, k_clusters + 1):
+        assert_eq(
+            dd.merge(empty_df, df[df.cluster == x], how="outer").compute(),
+            df[df.cluster == x].compute(),
+            check_index=False,
+        )