Repository: dask/dask

Issue:
Unable to append dataframe using .to_parquet if column contains a dictionary/structure
**What happened**: Saving a dataframe using `.to_parquet(path, append=True)` fails when one of the columns contains a dictionary. Raises: `ValueError: Appended columns not the same.`

**What you expected to happen**: Expected to be able to append parquet datasets (this works if using simple column types).

**Minimal Complete Verifiable Example**:

```python
import dask.dataframe as dd
import pandas as pd

dts = pd.date_range('2020-01-01', '2021-01-01')
df = pd.DataFrame(
    {'value': [{'x': x} for x in range(len(dts))]},
    index=dts
)
print(df.head())

ddf = dd.from_pandas(df, npartitions=1)

# This works:
ddf.to_parquet('/tmp/test_parquet', append=True)

# Can successfully read data
print(dd.read_parquet('/tmp/test_parquet').head())

# This fails:
ddf.to_parquet('/tmp/test_parquet', append=True)
# ValueError: Appended columns not the same.
# Previous: ['x', '__null_dask_index__'] | New: ['__null_dask_index__', 'value']
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2021.03.1
- Python version: 3.7.10
- Operating System: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Install method (conda, pip, source): pip (using Google Colab)



Discussion:
Thanks for raising an issue @toby-coleman. Are you using `fastparquet` or `pyarrow`? Also what version of the parquet engine library are you using? 

cc @dask/io 
I'm using `pyarrow` version `3.0.0`.
You may want to try with v2.x for now
Just tried it on `2.0.0` and I get the same thing.
Thanks for raising this @toby-coleman - I don't think there is any test coverage for dictionary-column types, so I am not surprised that there are issues. I don’t think this is working for me with fastparquet or pyarrow locally, but the pyarrow failure seems like it should be easy to address.

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -750,14 +750,14 @@ def initialize_write(
                 # Original dataset does not exist - cannot append
                 append = False
         if append:
-            names = dataset.metadata.schema.names
+            arrow_schema = dataset.schema.to_arrow_schema()
+            names = arrow_schema.names
             has_pandas_metadata = (
-                dataset.schema.to_arrow_schema().metadata is not None
-                and b"pandas" in dataset.schema.to_arrow_schema().metadata
+                arrow_schema.metadata is not None and b"pandas" in arrow_schema.metadata
             )
             if has_pandas_metadata:
                 pandas_metadata = json.loads(
-                    dataset.schema.to_arrow_schema().metadata[b"pandas"].decode("utf8")
+                    arrow_schema.metadata[b"pandas"].decode("utf8")
                 )
                 categories = [
                     c["name"]
@@ -766,7 +766,7 @@ def initialize_write(
                 ]
             else:
                 categories = None
-            dtypes = _get_pyarrow_dtypes(dataset.schema.to_arrow_schema(), categories)
+            dtypes = _get_pyarrow_dtypes(arrow_schema, categories)
             if set(names) != set(df.columns) - set(partition_on):
                 raise ValueError(
                     "Appended columns not the same.\n"


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -835,6 +835,35 @@ def test_append_different_columns(tmpdir, engine):
     assert "Appended dtypes" in str(excinfo.value)
 
 
+def test_append_dict_column(tmpdir, engine):
+    # See: https://github.com/dask/dask/issues/7492
+
+    if engine == "fastparquet":
+        pytest.xfail("Fastparquet engine is missing dict-column support")
+    elif pa.__version__ < LooseVersion("1.0.1"):
+        pytest.skip("Newer PyArrow version required for dict-column support.")
+
+    tmp = str(tmpdir)
+    dts = pd.date_range("2020-01-01", "2021-01-01")
+    df = pd.DataFrame(
+        {"value": [{"x": x} for x in range(len(dts))]},
+        index=dts,
+    )
+    ddf1 = dd.from_pandas(df, npartitions=1)
+
+    # Write ddf1 to tmp, and then append it again
+    ddf1.to_parquet(tmp, append=True, engine=engine)
+    ddf1.to_parquet(tmp, append=True, engine=engine, ignore_divisions=True)
+
+    # Read back all data (ddf1 + ddf1)
+    ddf2 = dd.read_parquet(tmp, engine=engine)
+
+    # Check computed result
+    expect = pd.concat([df, df])
+    result = ddf2.compute()
+    assert_eq(expect, result)
+
+
 @write_read_engines_xfail
 def test_ordering(tmpdir, write_engine, read_engine):
     tmp = str(tmpdir)