Repository: dask/dask
Pull request: Add ignore_order parameter in dd.concat (#7398)
- Closes #7398
- Tests added / passed
- Passes `black dask` / `flake8 dask`


Problem statement:
add "ignore_order" parameter for categoricals in dd.concat
it would be handy to have a passthrough parameter to dd.concat  - currently the union_categoricals call is parameter-free:

```
    dtype = "category"
            if left is not None and right is not None:
                dtype = union_categoricals(
                    [left.astype("category").values, right.astype("category").values]
                ).dtype
```

This can be made more flexible as follows from the pandas docs ( I believe, though I know there are details to how dask is handling categoricals vs pandas )

```
In [1]: a = pd.Categorical(["a", "b"], ordered=True)
In [2]: b = pd.Categorical(["a", "b", "c"], ordered=True)
In [3]: union_categoricals([a, b])
Out[3]:
TypeError: to union ordered Categoricals, all categories must be the same
```

```
In [171]: a = pd.Categorical(["a", "b", "c"], ordered=True)
In [172]: b = pd.Categorical(["c", "b", "a"], ordered=True)
In [173]: union_categoricals([a, b], ignore_order=True)
Out[173]: 
[a, b, c, c, b, a]
Categories (3, object): [a, b, c]
```

Hints from issue discussion:
That sounds like a nice addition, thanks for writing it up! If you have time to work on a pull request, that is the best and quickest way to get this feature :)
Can i work on this ?
I see two options - 

```
@concat_dispatch.register((pd.DataFrame, pd.Series, pd.Index))                                                                                                                                                                                    
def concat_pandas(                                                                                                                                                                                                                                
    dfs,                                                                                                                                                                                                                                          
    axis=0,                                                                                                                                                                                                                                       
    join="outer",                                                                                                                                                                                                                                 
    uniform=False,                                                                                                                                                                                                                                
    filter_warning=True,                                                                                                                                                                                                                          
    ignore_index=False,                                                                                                                                                                                                                           
    **kwargs                                                                                                                                                                                                                                      
):    
```

Here kwargs is passed through, and so we could add a check -

```
ignore_order = kwargs['ignore_order'] if 'ignore_order' in kwargs else False
```

where False is the pandas default.

Or alternatively, we explicitly daisy chain the ignore_order argument down through the functions and use directly.

Thoughts?


@gsc2001 Are you working on this?
Anyone can work on this, no need to ask. Just open a PR that references this issue.

Commit message: Add ignore_order parameter in dd.concat (#7398)

Patch to review:
diff --git a/dask/dataframe/methods.py b/dask/dataframe/methods.py
index 7c47d443c9a..2798c099c61 100644
--- a/dask/dataframe/methods.py
+++ b/dask/dataframe/methods.py
@@ -398,6 +398,7 @@ def concat(
     join="outer",
     uniform=False,
     filter_warning=True,
+    ignore_order=False,
     ignore_index=False,
     **kwargs
 ):
@@ -416,8 +417,11 @@ def concat(
         True if all arguments have the same columns and dtypes (but not
         necessarily categories). Default is False.
     ignore_index : bool, optional
-        Whether to allow index values to be ignored/droped during
+        Whether to allow index values to be ignored/dropped during
         concatenation. Default is False.
+    ignore_order : bool, optional
+        Whether to ignore the order when doing the union of categoricals.
+        Default is False.
     """
     if len(dfs) == 1:
         return dfs[0]
@@ -430,6 +434,7 @@ def concat(
             uniform=uniform,
             filter_warning=filter_warning,
             ignore_index=ignore_index,
+            ignore_order=ignore_order,
             **kwargs
         )
 
@@ -441,6 +446,7 @@ def concat_pandas(
     join="outer",
     uniform=False,
     filter_warning=True,
+    ignore_order=False,
     ignore_index=False,
     **kwargs
 ):
@@ -453,7 +459,9 @@ def concat_pandas(
             for i in range(1, len(dfs)):
                 if not isinstance(dfs[i], pd.CategoricalIndex):
                     dfs[i] = dfs[i].astype("category")
-            return pd.CategoricalIndex(union_categoricals(dfs), name=dfs[0].name)
+            return pd.CategoricalIndex(
+                union_categoricals(dfs, ignore_order=ignore_order), name=dfs[0].name
+            )
         elif isinstance(dfs[0], pd.MultiIndex):
             first, rest = dfs[0], dfs[1:]
             if all(
@@ -545,7 +553,7 @@ def concat_pandas(
                             codes, sample.cat.categories, sample.cat.ordered
                         )
                         parts.append(data)
-                out[col] = union_categoricals(parts)
+                out[col] = union_categoricals(parts, ignore_order=ignore_order)
                 # Pandas resets index type on assignment if frame is empty
                 # https://github.com/pandas-dev/pandas/issues/17101
                 if not len(temp_ind):
@@ -562,7 +570,11 @@ def concat_pandas(
         if is_categorical_dtype(dfs2[0].dtype):
             if ind is None:
                 ind = concat([df.index for df in dfs2])
-            return pd.Series(union_categoricals(dfs2), index=ind, name=dfs2[0].name)
+            return pd.Series(
+                union_categoricals(dfs2, ignore_order=ignore_order),
+                index=ind,
+                name=dfs2[0].name,
+            )
         with warnings.catch_warnings():
             if filter_warning:
                 warnings.simplefilter("ignore", FutureWarning)
diff --git a/dask/dataframe/multi.py b/dask/dataframe/multi.py
index dc472567fb1..6e925931fab 100644
--- a/dask/dataframe/multi.py
+++ b/dask/dataframe/multi.py
@@ -956,31 +956,38 @@ def merge_asof(
 ###############################################################
 
 
-def concat_and_check(dfs):
+def concat_and_check(dfs, ignore_order=False):
     if len(set(map(len, dfs))) != 1:
         raise ValueError("Concatenated DataFrames of different lengths")
-    return methods.concat(dfs, axis=1)
+    return methods.concat(dfs, ignore_order=ignore_order, axis=1)
 
 
-def concat_unindexed_dataframes(dfs, **kwargs):
+def concat_unindexed_dataframes(dfs, ignore_order=False, **kwargs):
     name = "concat-" + tokenize(*dfs)
 
     dsk = {
-        (name, i): (concat_and_check, [(df._name, i) for df in dfs])
+        (name, i): (concat_and_check, [(df._name, i) for df in dfs], ignore_order)
         for i in range(dfs[0].npartitions)
     }
 
-    meta = methods.concat([df._meta for df in dfs], axis=1, **kwargs)
+    meta = methods.concat(
+        [df._meta for df in dfs], axis=1, ignore_order=ignore_order, **kwargs
+    )
 
     graph = HighLevelGraph.from_collections(name, dsk, dependencies=dfs)
     return new_dd_object(graph, name, meta, dfs[0].divisions)
 
 
-def concat_indexed_dataframes(dfs, axis=0, join="outer", **kwargs):
+def concat_indexed_dataframes(dfs, axis=0, join="outer", ignore_order=False, **kwargs):
     """ Concatenate indexed dataframes together along the index """
     warn = axis != 0
     meta = methods.concat(
-        [df._meta for df in dfs], axis=axis, join=join, filter_warning=warn, **kwargs
+        [df._meta for df in dfs],
+        axis=axis,
+        join=join,
+        filter_warning=warn,
+        ignore_order=ignore_order,
+        **kwargs,
     )
     empties = [strip_unknown_categories(df._meta) for df in dfs]
 
@@ -997,7 +1004,10 @@ def concat_indexed_dataframes(dfs, axis=0, join="outer", **kwargs):
     uniform = False
 
     dsk = dict(
-        ((name, i), (methods.concat, part, axis, join, uniform, filter_warning))
+        (
+            (name, i),
+            (methods.concat, part, axis, join, uniform, filter_warning, ignore_order),
+        )
         for i, part in enumerate(parts2)
     )
     for df in dfs2:
@@ -1006,13 +1016,17 @@ def concat_indexed_dataframes(dfs, axis=0, join="outer", **kwargs):
     return new_dd_object(dsk, name, meta, divisions)
 
 
-def stack_partitions(dfs, divisions, join="outer", **kwargs):
+def stack_partitions(dfs, divisions, join="outer", ignore_order=False, **kwargs):
     """Concatenate partitions on axis=0 by doing a simple stack"""
     # Use _meta_nonempty as pandas.concat will incorrectly cast float to datetime
     # for empty data frames. See https://github.com/pandas-dev/pandas/issues/32934.
     meta = make_meta(
         methods.concat(
-            [df._meta_nonempty for df in dfs], join=join, filter_warning=False, **kwargs
+            [df._meta_nonempty for df in dfs],
+            join=join,
+            filter_warning=False,
+            ignore_order=ignore_order,
+            **kwargs,
         )
     )
     empty = strip_unknown_categories(meta)
@@ -1054,11 +1068,22 @@ def stack_partitions(dfs, divisions, join="outer", **kwargs):
         except (ValueError, TypeError):
             match = False
 
+        filter_warning = True
+        uniform = False
+
         for key in df.__dask_keys__():
             if match:
                 dsk[(name, i)] = key
             else:
-                dsk[(name, i)] = (methods.concat, [empty, key], 0, join)
+                dsk[(name, i)] = (
+                    methods.concat,
+                    [empty, key],
+                    0,
+                    join,
+                    uniform,
+                    filter_warning,
+                    ignore_order,
+                )
             i += 1
 
     return new_dd_object(dsk, name, meta, divisions)
@@ -1070,6 +1095,7 @@ def concat(
     join="outer",
     interleave_partitions=False,
     ignore_unknown_divisions=False,
+    ignore_order=False,
     **kwargs,
 ):
     """Concatenate DataFrames along rows.
@@ -1103,6 +1129,8 @@ def concat(
     ignore_unknown_divisions : bool, default False
         By default a warning is raised if any input has unknown divisions.
         Set to True to disable this warning.
+    ignore_order : bool, default False
+        Wether to ignore order when doing the union of categoricals.
 
     Notes
     -----
@@ -1180,7 +1208,9 @@ def concat(
 
     if axis == 1:
         if all(df.known_divisions for df in dasks):
-            return concat_indexed_dataframes(dfs, axis=axis, join=join, **kwargs)
+            return concat_indexed_dataframes(
+                dfs, axis=axis, join=join, ignore_order=ignore_order, **kwargs
+            )
         elif (
             len(dasks) == len(dfs)
             and all(not df.known_divisions for df in dfs)
@@ -1193,7 +1223,7 @@ def concat(
                     " are \n aligned. This assumption is not generally "
                     "safe."
                 )
-            return concat_unindexed_dataframes(dfs, **kwargs)
+            return concat_unindexed_dataframes(dfs, ignore_order=ignore_order, **kwargs)
         else:
             raise ValueError(
                 "Unable to concatenate DataFrame with unknown "
@@ -1211,15 +1241,23 @@ def concat(
                     # remove last to concatenate with next
                     divisions += df.divisions[:-1]
                 divisions += dfs[-1].divisions
-                return stack_partitions(dfs, divisions, join=join, **kwargs)
+                return stack_partitions(
+                    dfs, divisions, join=join, ignore_order=ignore_order, **kwargs
+                )
             elif interleave_partitions:
-                return concat_indexed_dataframes(dfs, join=join, **kwargs)
+                return concat_indexed_dataframes(
+                    dfs, join=join, ignore_order=ignore_order, **kwargs
+                )
             else:
                 divisions = [None] * (sum([df.npartitions for df in dfs]) + 1)
-                return stack_partitions(dfs, divisions, join=join, **kwargs)
+                return stack_partitions(
+                    dfs, divisions, join=join, ignore_order=ignore_order, **kwargs
+                )
         else:
             divisions = [None] * (sum([df.npartitions for df in dfs]) + 1)
-            return stack_partitions(dfs, divisions, join=join, **kwargs)
+            return stack_partitions(
+                dfs, divisions, join=join, ignore_order=ignore_order, **kwargs
+            )
 
 
 def _contains_index_name(df, columns_or_index):
diff --git a/dask/dataframe/tests/test_multi.py b/dask/dataframe/tests/test_multi.py
index 5bd8e108cca..d4f5ddc09b1 100644
--- a/dask/dataframe/tests/test_multi.py
+++ b/dask/dataframe/tests/test_multi.py
@@ -2143,6 +2143,25 @@ def test_groupby_concat_cudf(engine):
     assert_eq(res_dd.compute().sort_index(), res.sort_index())
 
 
+def test_concat_ignore_order():
+    pdf1 = pd.DataFrame(
+        {
+            "x": pd.Categorical(
+                ["a", "b", "c", "a"], categories=["a", "b", "c"], ordered=True
+            )
+        }
+    )
+    ddf1 = dd.from_pandas(pdf1, 2)
+    pdf2 = pd.DataFrame(
+        {"x": pd.Categorical(["c", "b", "a"], categories=["c", "b", "a"], ordered=True)}
+    )
+    ddf2 = dd.from_pandas(pdf2, 2)
+    expected = pd.concat([pdf1, pdf2])
+    expected["x"] = expected["x"].astype("category")
+    result = dd.concat([ddf1, ddf2], ignore_order=True)
+    assert_eq(result, expected)
+
+
 def test_categorical_join():
     # https://github.com/dask/dask/issues/6134
     df = pd.DataFrame(


Code review comments:
diff_hunk: @@ -1103,6 +1129,8 @@ def concat(
     ignore_unknown_divisions : bool, default False
         By default a warning is raised if any input has unknown divisions.
         Set to True to disable this warning.
+    ignore_order : bool, default False
+        Wether to ignore order when doing the union of categoricals.
path: dask/dataframe/multi.py
text: @user1:
```suggestion
        Whether to ignore order when doing the union of categoricals.
```