Repository: dask/dask

Issue:
get_dummies error: order of columns does not match
**What happened**:
When applying `categorize` and then `get_dummies` to columns that originally had a numerical data type, in some cases `get_dummies` fails with
```
ValueError: The columns in the computed data do not match the columns in the provided metadata
Order of columns does not match
```

I could not find the exact pattern behind these failures (happens on real-world data only on some columns, with no clear pattern), but I managed to produce a tiny example that exhibits the issue.

**What you expected to happen**:
`get_dummies` works without exceptions.

**Minimal Complete Verifiable Example**:

```python
import pandas as pd
import dask.dataframe as dd

data = pd.DataFrame(data={
    'a': [0.0, 0.0, 1.0, 1.0, 0.0],
    'b': [1.0, 0.0, 1.0, 0.0, 1.0]
})

data = dd.from_pandas(data, npartitions=2)

CAT = ['a', 'b']
data[CAT] = data[CAT].categorize(CAT)

dd.get_dummies(data[CAT]).compute()
```

**Traceback**:
```
Traceback (most recent call last):
  File "<redacted>/src/test_dd.py", line 16, in <module>
    dd.get_dummies(data[CAT]).compute()
  File "<redacted>/lib/python3.9/site-packages/dask/base.py", line 292, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "<redacted>/lib/python3.9/site-packages/dask/base.py", line 575, in compute
    results = schedule(dsk, keys, **kwargs)
  File "<redacted>/lib/python3.9/site-packages/dask/threaded.py", line 81, in get
    results = get_async(
  File "<redacted>/lib/python3.9/site-packages/dask/local.py", line 508, in get_async
    raise_exception(exc, tb)
  File "<redacted>/lib/python3.9/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "<redacted>/lib/python3.9/site-packages/dask/local.py", line 221, in execute_task
    result = _execute_task(task, data)
  File "<redacted>/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "<redacted>/lib/python3.9/site-packages/dask/optimization.py", line 990, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "<redacted>/lib/python3.9/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
  File "<redacted>/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "<redacted>/lib/python3.9/site-packages/dask/utils.py", line 40, in apply
    return func(*args, **kwargs)
  File "<redacted>/lib/python3.9/site-packages/dask/dataframe/core.py", line 6436, in apply_and_enforce
    check_matching_columns(meta, df)
  File "<redacted>/lib/python3.9/site-packages/dask/dataframe/utils.py", line 415, in check_matching_columns
    raise ValueError(
ValueError: The columns in the computed data do not match the columns in the provided metadata
Order of columns does not match
```

**Environment**:

- Dask version: 2022.05.00
- Python version: 3.9.12
- Operating System: Ubuntu 20.04
- Install method (conda, pip, source): conda


Discussion:

Resolving patch:
diff --git a/dask/dataframe/categorical.py b/dask/dataframe/categorical.py
--- a/dask/dataframe/categorical.py
+++ b/dask/dataframe/categorical.py
@@ -153,6 +153,9 @@ def categorize(df, columns=None, index=None, split_every=None, **kwargs):
         df.__class__, dsk, (prefix, 0), **kwargs
     )
 
+    # some operations like get_dummies() rely on the order of categories
+    categories = {k: v.sort_values() for k, v in categories.items()}
+
     # Categorize each partition
     return df.map_partitions(_categorize_block, categories, index)
 


Test patch:
diff --git a/dask/dataframe/tests/test_reshape.py b/dask/dataframe/tests/test_reshape.py
--- a/dask/dataframe/tests/test_reshape.py
+++ b/dask/dataframe/tests/test_reshape.py
@@ -29,6 +29,15 @@ def test_get_dummies(data):
     tm.assert_index_equal(res.columns, exp.columns)
 
 
+def test_get_dummies_categories_order():
+    df = pd.DataFrame({"a": [0.0, 0.0, 1.0, 1.0, 0.0], "b": [1.0, 0.0, 1.0, 0.0, 1.0]})
+    ddf = dd.from_pandas(df, npartitions=1)
+    ddf = ddf.categorize(columns=["a", "b"])
+    res_p = pd.get_dummies(df.astype("category"))
+    res_d = dd.get_dummies(ddf)
+    assert_eq(res_d, res_p)
+
+
 def test_get_dummies_object():
     df = pd.DataFrame(
         {