Repository: dask/dask

Issue:
[FEA] Generalize concat_unindexed_dataframes to support non-pandas objects
Following up on #5597, if `corr` ends up being called on GPU dataframes without known divisions, we go down a codepath in `concat` that still enforces a pandas concat due to `concat_unindexed_dataframes`.

https://github.com/dask/dask/blob/3960c6518318f2417658c2fc47cd5b5ece726f8b/dask/dataframe/multi.py#L1046-L1060

https://github.com/dask/dask/blob/3960c6518318f2417658c2fc47cd5b5ece726f8b/dask/dataframe/multi.py#L868-L879

One possible solution to this is to generalize this concatenation to support non-pandas objects.




Discussion:
Thanks for raising @beckernick  - using `methods.concat` should help us here.  Do you have a specific cudf example I can use to check that the fix covers your workflow?
I suspect that this would cover the workflow (though rapidsai/cudf#3363 may be hiding some additional work)

```python
import cudf
import dask_cudf
import dask.dataframe as dd

d1 = cudf.datasets.randomdata(100, dtypes={"a":float, "b":float, "f":int})
d2 = cudf.datasets.randomdata(100, dtypes={"c":float, "d":float, "q":int})
dd1 = dask_cudf.from_cudf(d1, 5)
dd2 = dask_cudf.from_cudf(d2, 5)

# groupby-agg to force known_divisions to be False
grouped_dd1 = dd1.groupby(['a']).sum()
grouped_dd2 = dd2.groupby(['c']).sum()

res = dd.concat([grouped_dd1, grouped_dd2], axis=1)
```
Just a note that this error should be resolved by using `methods.concat` on [this line](https://github.com/dask/dask/blob/3960c6518318f2417658c2fc47cd5b5ece726f8b/dask/dataframe/multi.py#L865) and [this line](https://github.com/dask/dask/blob/3960c6518318f2417658c2fc47cd5b5ece726f8b/dask/dataframe/multi.py#L876).  However, I was testing the behavior and noticed that the cudf backend seems to be doing an incorrect `concat` operation (for `axis=1`):


```python
In [1]: import pandas as pd 
   ...: import cudf                                                                                                                                                                           

In [2]: d1 = cudf.datasets.randomdata(3, dtypes={"a":float, "b":float, "f":int}) 
   ...: d2 = cudf.datasets.randomdata(3, dtypes={"c":float, "d":float, "q":int}) 
   ...: gbd1 = d1.groupby(['a']).sum() 
   ...: gbd2 = d2.groupby(['c']).sum()                                                                                                                                                        

In [3]: pd1 = d1.to_pandas() 
   ...: pd2 = d2.to_pandas() 
   ...: gbpd1 = pd1.groupby(['a']).sum() 
   ...: gbpd2 = pd2.groupby(['c']).sum()                                                                                                                                                      

In [4]: cudf.concat([gbd1, gbd2], axis=1)                                                                                                                                                     
Out[4]: 
                  b     f         d     q
a                                        
-0.633849 -0.697526   951 -0.247623  1024
-0.133484 -0.383614  1019  0.787768  1018
 0.166028  0.716850  1022  0.051626  1036

In [5]: pd.concat([gbpd1, gbpd2], axis=1)                                                                                                                                                     
Out[5]: 
                  b       f         d       q
-0.633849 -0.697526   951.0       NaN     NaN
-0.411569       NaN     NaN -0.247623  1024.0
-0.133484 -0.383614  1019.0       NaN     NaN
 0.166028  0.716850  1022.0       NaN     NaN
 0.910727       NaN     NaN  0.787768  1018.0
 0.930635       NaN     NaN  0.051626  1036.0
```

Resolving patch:
diff --git a/dask/dataframe/multi.py b/dask/dataframe/multi.py
--- a/dask/dataframe/multi.py
+++ b/dask/dataframe/multi.py
@@ -862,7 +862,7 @@ def merge_asof(
 def concat_and_check(dfs):
     if len(set(map(len, dfs))) != 1:
         raise ValueError("Concatenated DataFrames of different lengths")
-    return pd.concat(dfs, axis=1)
+    return methods.concat(dfs, axis=1)
 
 
 def concat_unindexed_dataframes(dfs):
@@ -873,7 +873,7 @@ def concat_unindexed_dataframes(dfs):
         for i in range(dfs[0].npartitions)
     }
 
-    meta = pd.concat([df._meta for df in dfs], axis=1)
+    meta = methods.concat([df._meta for df in dfs], axis=1)
 
     graph = HighLevelGraph.from_collections(name, dsk, dependencies=dfs)
     return new_dd_object(graph, name, meta, dfs[0].divisions)


Test patch:
diff --git a/dask/dataframe/tests/test_multi.py b/dask/dataframe/tests/test_multi.py
--- a/dask/dataframe/tests/test_multi.py
+++ b/dask/dataframe/tests/test_multi.py
@@ -1997,3 +1997,51 @@ def test_dtype_equality_warning():
         dd.multi.warn_dtype_mismatch(df1, df2, "a", "a")
 
     assert len(r) == 0
+
+
+@pytest.mark.parametrize("engine", ["pandas", "cudf"])
+def test_groupby_concat_cudf(engine):
+
+    # NOTE: Issue #5643 Reproducer
+
+    size = 6
+    npartitions = 3
+    d1 = pd.DataFrame(
+        {
+            "a": np.random.permutation(np.arange(size)),
+            "b": np.random.randint(100, size=size),
+        }
+    )
+    d2 = pd.DataFrame(
+        {
+            "c": np.random.permutation(np.arange(size)),
+            "d": np.random.randint(100, size=size),
+        }
+    )
+
+    if engine == "cudf":
+        # NOTE: engine == "cudf" requires cudf/dask_cudf,
+        # will be skipped by non-GPU CI.
+
+        cudf = pytest.importorskip("cudf")
+        dask_cudf = pytest.importorskip("dask_cudf")
+
+        d1 = cudf.from_pandas(d1)
+        d2 = cudf.from_pandas(d2)
+        dd1 = dask_cudf.from_cudf(d1, npartitions)
+        dd2 = dask_cudf.from_cudf(d2, npartitions)
+    else:
+        dd1 = dd.from_pandas(d1, npartitions)
+        dd2 = dd.from_pandas(d2, npartitions)
+
+    grouped_d1 = d1.groupby(["a"]).sum()
+    grouped_d2 = d2.groupby(["c"]).sum()
+    res = concat([grouped_d1, grouped_d2], axis=1)
+
+    grouped_dd1 = dd1.groupby(["a"]).sum()
+    grouped_dd2 = dd2.groupby(["c"]).sum()
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore", UserWarning)
+        res_dd = dd.concat([grouped_dd1, grouped_dd2], axis=1)
+
+    assert_eq(res_dd.compute().sort_index(), res.sort_index())