Repository: dask/dask

Issue:
Empty partitions failure
The to_parquet function fails using the pyarrow engine if you have a dask dataframe with an empty partition. It may require a certain mix of data types to trigger the defect.

Support you have a csv file:
field1, field2
a,5

you then read it in, repartition to 2 partitions and then try to write as parquet.

        import dask.dataframe as dd
        ddf = dd.read_parquet(<path to csv>)
        ddf2 = ddf.repartition(npartitions=ddf.napartitions*2)
        ddf2.to_parquet(<path>, engine='pyarrow')

This will fail with a "inconsistent schema" error. I can resolve this error by using the following function to first remove the empty dataframes; the function comes from MRocklin in 2017...

          def cull_empty_partitions(ddf):
              ll = list(ddf.map_partitions(len).compute())
              df_delayed = ddf.to_delayed()
              df_delayed_new = list()
              pempty = None
              for ix, n in enumerate(ll):
                  if 0 == n:
                      pempty = ddf.get_partition(ix)
                  else:
                      df_delayed_new.append(df_delayed[ix])
              if pempty is not None:
                  ddf = dd.from_delayed(df_delayed_new, meta=pempty)
              return ddf

dask version: 2021.12.0
python version: 3.9
OS version: MacOS and Windows tested
install method: conda




Discussion:
When I run the reproducer (with ~typos fixed~ the first `read_parquet` changed to `read_csv`), I get:

```
`RuntimeError: Schemas are inconsistent, try using `to_parquet(..., schema="infer")`, or pass an explicit pyarrow schema. Such as `to_parquet(..., schema={"column1": pa.string()})`
```

If I follow the advice in the warning, and use ~`ddf2.to_parquet(<path>, engine='pyarrow')`~ `ddf2.to_parquet(<path>, engine='pyarrow', schema='infer')`, everything runs fine for me.  Are you not getting this warning, or did it not resolve your issue?
Apologies for whatever the typos were!
Yes, you can follow the suggestion; I assume you used ddf2.to_parquet(<path>, engine='pyarrow', schema='infer')?
This might work, but should not be necessary; it is not necessary if you do not have an empty partition. I believe it is only occurring because of the empty partition, and this would therefore be unintended behaviour.

MRocklin asked for reports of functions that do not work on data dataframes that contain empty partitions. 
>I assume you used ddf2.to_parquet(, engine='pyarrow', schema='infer')?

Yes. My mistake!

>This might work, but should not be necessary; it is not necessary if you do not have an empty partition. I believe it is only occurring because of the empty partition, and this would therefore be unintended behaviour.

I agree that it would be ideal for this to work with an empty partition, but the problem has come up before and the only way to "fix" it is to change the default behavior to `schema="infer"`. Since this blind schema inference can be expensive, the compromise was a clear error message (see [4194](https://github.com/dask/dask/issues/4194#issuecomment-488694357), #6490, and #6657).

Note that there is also another open issue asking to further improve this error message (#8619), but I don't think there is any way to avoid making the user pass a `schema` argument in Dask.  (but we are certainly open to suggestions).
@rjzamora, perhaps a naive question, but could we just not write any file for an empty partition? This would mean that a `part.#.parquet` file wouldn't exist for that partition, but would avoid doing any schema merging. I suppose this would look confusing to downstream consumers (like, perhaps a file is missing), but it would avoid the issue I think.
>@rjzamora, perhaps a naive question, but could we just not write any file for an empty partition? This would mean that a part.#.parquet file wouldn't exist for that partition, but would avoid doing any schema merging. I suppose this would look confusing to downstream consumers (like, perhaps a file is missing), but it would avoid the issue I think.

This is probably fine. I was actually overlooking this possibility, because I was thinking about the case of "null" partitions in addition to "empty" partitions.  However, if the partition is completely empty, then I don't see any obvious problem with your suggestion (although, we may want it to be optional?).
Alternatively, is this mainly/only an issue when writing the `_metadata` file? If we switched to defaulting to `write_metadata_file=False` would that resolve this issue for most users? I know that `_meta` doesn't map 1:1 to a pyarrow schema, but if empty files were close-enough in schema (e.g. all columns present, proper int/float/str/... dtypes) would that be enough that most downstream readers (dask & others) be fine? 
>Alternatively, is this mainly/only an issue when writing the _metadata file? If we switched to defaulting to write_metadata_file=False would that resolve this issue for most users? I know that _meta doesn't map 1:1 to a pyarrow schema, but if empty files were close-enough in schema (e.g. all columns present, proper int/float/str/... dtypes) would that be enough that most downstream readers (dask & others) be fine?

I've been thinking about this a bit, and will probably need to summarize my thoughts in #8901 - However, the short version is that I would love to rip out _metadata usage from both a maintenance and stability perspective, but the clear performance benefit of using _metadata still makes me a bit hesitant to suggest this (though I am leaning toward it).

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -1,4 +1,5 @@
 import json
+import textwrap
 from collections import defaultdict
 from datetime import datetime
 
@@ -29,7 +30,7 @@
     _open_input_files,
 )
 from dask.dataframe.utils import clear_known_categories
-from dask.delayed import Delayed, delayed
+from dask.delayed import Delayed
 from dask.utils import getargspec, natural_sort_key
 
 # Check PyArrow version for feature support
@@ -504,50 +505,26 @@ def initialize_write(
         partition_on=None,
         ignore_divisions=False,
         division_info=None,
-        schema=None,
+        schema="infer",
         index_cols=None,
         **kwargs,
     ):
-        # Infer schema if "infer"
-        # (also start with inferred schema if user passes a dict)
         if schema == "infer" or isinstance(schema, dict):
-
             # Start with schema from _meta_nonempty
-            _schema = pa.Schema.from_pandas(
+            inferred_schema = pa.Schema.from_pandas(
                 df._meta_nonempty.set_index(index_cols)
                 if index_cols
                 else df._meta_nonempty
-            )
+            ).remove_metadata()
 
             # Use dict to update our inferred schema
             if isinstance(schema, dict):
                 schema = pa.schema(schema)
                 for name in schema.names:
-                    i = _schema.get_field_index(name)
+                    i = inferred_schema.get_field_index(name)
                     j = schema.get_field_index(name)
-                    _schema = _schema.set(i, schema.field(j))
-
-            # If we have object columns, we need to sample partitions
-            # until we find non-null data for each column in `sample`
-            sample = [col for col in df.columns if df[col].dtype == "object"]
-            if sample and schema == "infer":
-                delayed_schema_from_pandas = delayed(pa.Schema.from_pandas)
-                for i in range(df.npartitions):
-                    # Keep data on worker
-                    _s = delayed_schema_from_pandas(
-                        df[sample].to_delayed()[i]
-                    ).compute()
-                    for name, typ in zip(_s.names, _s.types):
-                        if typ != "null":
-                            i = _schema.get_field_index(name)
-                            j = _s.get_field_index(name)
-                            _schema = _schema.set(i, _s.field(j))
-                            sample.remove(name)
-                    if not sample:
-                        break
-
-            # Final (inferred) schema
-            schema = _schema
+                    inferred_schema = inferred_schema.set(i, schema.field(j))
+            schema = inferred_schema
 
         # Check that target directory exists
         fs.mkdirs(path, exist_ok=True)
@@ -651,10 +628,33 @@ def initialize_write(
     def _pandas_to_arrow_table(
         cls, df: pd.DataFrame, preserve_index=False, schema=None
     ) -> pa.Table:
-        table = pa.Table.from_pandas(
-            df, nthreads=1, preserve_index=preserve_index, schema=schema
-        )
-        return table
+        try:
+            return pa.Table.from_pandas(
+                df, nthreads=1, preserve_index=preserve_index, schema=schema
+            )
+        except pa.ArrowException as exc:
+            if schema is None:
+                raise
+            df_schema = pa.Schema.from_pandas(df)
+            expected = textwrap.indent(
+                schema.to_string(show_schema_metadata=False), "    "
+            )
+            actual = textwrap.indent(
+                df_schema.to_string(show_schema_metadata=False), "    "
+            )
+            raise ValueError(
+                f"Failed to convert partition to expected pyarrow schema:\n"
+                f"    `{exc!r}`\n"
+                f"\n"
+                f"Expected partition schema:\n"
+                f"{expected}\n"
+                f"\n"
+                f"Received partition schema:\n"
+                f"{actual}\n"
+                f"\n"
+                f"This error *may* be resolved by passing in schema information for\n"
+                f"the mismatched column(s) using the `schema` keyword in `to_parquet`."
+            ) from None
 
     @classmethod
     def write_partition(
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -610,7 +610,7 @@ def to_parquet(
     write_metadata_file=None,
     compute=True,
     compute_kwargs=None,
-    schema=None,
+    schema="infer",
     name_function=None,
     **kwargs,
 ):
@@ -667,20 +667,26 @@ def to_parquet(
         default), a ``_metadata`` file will only be written if ``append=True``
         and the dataset already has a ``_metadata`` file.
     compute : bool, default True
-        If :obj:`True` (default) then the result is computed immediately. If  :obj:`False`
-        then a ``dask.dataframe.Scalar`` object is returned for future computation.
+        If ``True`` (default) then the result is computed immediately. If
+        ``False`` then a ``dask.dataframe.Scalar`` object is returned for
+        future computation.
     compute_kwargs : dict, default True
         Options to be passed in to the compute method
-    schema : Schema object, dict, or {"infer", None}, default None
-        Global schema to use for the output dataset. Alternatively, a `dict`
-        of pyarrow types can be specified (e.g. `schema={"id": pa.string()}`).
-        For this case, fields excluded from the dictionary will be inferred
-        from `_meta_nonempty`.  If "infer", the first non-empty and non-null
-        partition will be used to infer the type for "object" columns. If
-        None (default), we let the backend infer the schema for each distinct
-        output partition. If the partitions produce inconsistent schemas,
-        pyarrow will throw an error when writing the shared _metadata file.
-        Note that this argument is ignored by the "fastparquet" engine.
+    schema : pyarrow.Schema, dict, "infer", or None, default "infer"
+        Global schema to use for the output dataset. Defaults to "infer", which
+        will infer the schema from the dask dataframe metadata. This is usually
+        sufficient for common schemas, but notably will fail for ``object``
+        dtype columns that contain things other than strings. These columns
+        will require an explicit schema be specified. The schema for a subset
+        of columns can be overridden by passing in a dict of column names to
+        pyarrow types (for example ``schema={"field": pa.string()}``); columns
+        not present in this dict will still be automatically inferred.
+        Alternatively, a full ``pyarrow.Schema`` may be passed, in which case
+        no schema inference will be done. Passing in ``schema=None`` will
+        disable the use of a global file schema - each written file may use a
+        different schema dependent on the dtypes of the corresponding
+        partition. Note that this argument is ignored by the "fastparquet"
+        engine.
     name_function : callable, default None
         Function to generate the filename for each output partition.
         The function should accept an integer (partition index) as input and
diff --git a/dask/dataframe/io/parquet/fastparquet.py b/dask/dataframe/io/parquet/fastparquet.py
--- a/dask/dataframe/io/parquet/fastparquet.py
+++ b/dask/dataframe/io/parquet/fastparquet.py
@@ -1114,7 +1114,7 @@ def initialize_write(
         partition_on=None,
         ignore_divisions=False,
         division_info=None,
-        schema=None,
+        schema="infer",
         object_encoding="utf8",
         index_cols=None,
         custom_metadata=None,


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -119,13 +119,6 @@ def write_read_engines(**kwargs):
     )
 
 
-pyarrow_fastparquet_msg = "pyarrow schema and pandas metadata may disagree"
-write_read_engines_xfail = write_read_engines(
-    **{
-        "xfail_pyarrow_fastparquet": pyarrow_fastparquet_msg,
-    }
-)
-
 if (
     fastparquet
     and fastparquet_version < parse_version("0.5")
@@ -136,6 +129,7 @@ def write_read_engines(**kwargs):
     # categorical columns when using fastparquet 0.4.x, but this was (accidentally)
     # fixed in fastparquet 0.5.0
     fp_pandas_msg = "pandas with fastparquet engine does not preserve index"
+    pyarrow_fastparquet_msg = "pyarrow schema and pandas metadata may disagree"
     fp_pandas_xfail = write_read_engines(
         **{
             "xfail_pyarrow_fastparquet": pyarrow_fastparquet_msg,
@@ -144,7 +138,6 @@ def write_read_engines(**kwargs):
         }
     )
 else:
-    fp_pandas_msg = "pandas with fastparquet engine does not preserve index"
     fp_pandas_xfail = write_read_engines()
 
 
@@ -208,7 +201,7 @@ def test_local(tmpdir, write_engine, read_engine, has_metadata):
 
 
 @pytest.mark.parametrize("index", [False, True])
-@write_read_engines_xfail
+@write_read_engines()
 def test_empty(tmpdir, write_engine, read_engine, index):
     fn = str(tmpdir)
     df = pd.DataFrame({"a": ["a", "b", "b"], "b": [4, 5, 6]})[:0]
@@ -224,10 +217,7 @@ def test_empty(tmpdir, write_engine, read_engine, index):
 @write_read_engines()
 def test_simple(tmpdir, write_engine, read_engine):
     fn = str(tmpdir)
-    if write_engine != "fastparquet":
-        df = pd.DataFrame({"a": [b"a", b"b", b"b"], "b": [4, 5, 6]})
-    else:
-        df = pd.DataFrame({"a": ["a", "b", "b"], "b": [4, 5, 6]})
+    df = pd.DataFrame({"a": ["a", "b", "b"], "b": [4, 5, 6]})
     df.set_index("a", inplace=True, drop=True)
     ddf = dd.from_pandas(df, npartitions=2)
     ddf.to_parquet(fn, engine=write_engine)
@@ -942,9 +932,13 @@ def test_append_dict_column(tmpdir, engine):
     )
     ddf1 = dd.from_pandas(df, npartitions=1)
 
+    schema = {"value": pa.struct([("x", pa.int32())])}
+
     # Write ddf1 to tmp, and then append it again
-    ddf1.to_parquet(tmp, append=True, engine=engine)
-    ddf1.to_parquet(tmp, append=True, engine=engine, ignore_divisions=True)
+    ddf1.to_parquet(tmp, append=True, engine=engine, schema=schema)
+    ddf1.to_parquet(
+        tmp, append=True, engine=engine, schema=schema, ignore_divisions=True
+    )
 
     # Read back all data (ddf1 + ddf1)
     ddf2 = dd.read_parquet(tmp, engine=engine)
@@ -955,7 +949,7 @@ def test_append_dict_column(tmpdir, engine):
     assert_eq(expect, result)
 
 
-@write_read_engines_xfail
+@write_read_engines()
 def test_ordering(tmpdir, write_engine, read_engine):
     tmp = str(tmpdir)
     df = pd.DataFrame(
@@ -966,10 +960,6 @@ def test_ordering(tmpdir, write_engine, read_engine):
     ddf = dd.from_pandas(df, npartitions=2)
     dd.to_parquet(ddf, tmp, engine=write_engine)
 
-    if read_engine == "fastparquet":
-        pf = fastparquet.ParquetFile(tmp)
-        assert pf.columns == ["myindex", "c", "a", "b"]
-
     ddf2 = dd.read_parquet(tmp, index="myindex", engine=read_engine)
     assert_eq(ddf, ddf2, check_divisions=False)
 
@@ -1004,7 +994,11 @@ def test_read_parquet_custom_columns(tmpdir, engine):
         (pd.DataFrame({"x": [3, 2, 1]}), {}, {}),
         (pd.DataFrame({"x": ["c", "a", "b"]}), {}, {}),
         (pd.DataFrame({"x": ["cc", "a", "bbb"]}), {}, {}),
-        (pd.DataFrame({"x": [b"a", b"b", b"c"]}), {"object_encoding": "bytes"}, {}),
+        (
+            pd.DataFrame({"x": [b"a", b"b", b"c"]}),
+            {"object_encoding": "bytes", "schema": {"x": pa.binary()} if pa else None},
+            {},
+        ),
         (
             pd.DataFrame({"x": pd.Categorical(["a", "b", "a"])}),
             {},
@@ -1255,7 +1249,7 @@ def test_to_parquet_pyarrow_w_inconsistent_schema_by_partition_succeeds_w_manual
 @PYARROW_MARK
 @pytest.mark.parametrize("index", [False, True])
 @pytest.mark.parametrize("schema", ["infer", "complex"])
-def test_pyarrow_schema_inference(tmpdir, index, engine, schema):
+def test_pyarrow_schema_inference(tmpdir, index, schema):
     if schema == "complex":
         schema = {"index": pa.string(), "amount": pa.int64()}
 
@@ -1284,10 +1278,43 @@ def test_pyarrow_schema_inference(tmpdir, index, engine, schema):
         df = dd.from_pandas(df, npartitions=2)
 
     df.to_parquet(tmpdir, engine="pyarrow", schema=schema)
-    df_out = dd.read_parquet(tmpdir, engine=engine, calculate_divisions=True)
+    df_out = dd.read_parquet(tmpdir, engine="pyarrow", calculate_divisions=True)
     assert_eq(df, df_out)
 
 
+@PYARROW_MARK
+def test_pyarrow_schema_mismatch_error(tmpdir):
+    df1 = pd.DataFrame({"x": [1, 2, 3], "y": [4.5, 6, 7]})
+    df2 = pd.DataFrame({"x": [4, 5, 6], "y": ["a", "b", "c"]})
+
+    ddf = dd.from_delayed(
+        [dask.delayed(df1), dask.delayed(df2)], meta=df1, verify_meta=False
+    )
+
+    with pytest.raises(ValueError) as rec:
+        ddf.to_parquet(str(tmpdir), engine="pyarrow")
+
+    msg = str(rec.value)
+    assert "Failed to convert partition to expected pyarrow schema" in msg
+    assert "y: double" in str(rec.value)
+    assert "y: string" in str(rec.value)
+
+
+@PYARROW_MARK
+def test_pyarrow_schema_mismatch_explicit_schema_none(tmpdir):
+    df1 = pd.DataFrame({"x": [1, 2, 3], "y": [4.5, 6, 7]})
+    df2 = pd.DataFrame({"x": [4, 5, 6], "y": ["a", "b", "c"]})
+    ddf = dd.from_delayed(
+        [dask.delayed(df1), dask.delayed(df2)], meta=df1, verify_meta=False
+    )
+    ddf.to_parquet(str(tmpdir), engine="pyarrow", schema=None)
+    res = dd.read_parquet(tmpdir, engine="pyarrow")
+    sol = pd.concat([df1, df2])
+    # Only checking that the data was written correctly, we don't care about
+    # the incorrect _meta from read_parquet
+    assert_eq(res, sol, check_dtype=False)
+
+
 def test_partition_on(tmpdir, engine):
     tmpdir = str(tmpdir)
     df = pd.DataFrame(
@@ -2259,7 +2286,6 @@ def test_append_cat_fp(tmpdir, engine):
         pd.DataFrame({"x": [4, 5, 6, 1, 2, 3]}),
         pd.DataFrame({"x": ["c", "a", "b"]}),
         pd.DataFrame({"x": ["cc", "a", "bbb"]}),
-        pd.DataFrame({"x": [b"a", b"b", b"c"]}),
         pytest.param(pd.DataFrame({"x": pd.Categorical(["a", "b", "a"])})),
         pytest.param(pd.DataFrame({"x": pd.Categorical([1, 2, 1])})),
         pd.DataFrame({"x": list(map(pd.Timestamp, [3000000, 2000000, 1000000]))}),  # ms
@@ -3668,7 +3694,7 @@ def test_roundtrip_decimal_dtype(tmpdir):
     ]
     ddf1 = dd.from_pandas(pd.DataFrame(data), npartitions=1)
 
-    ddf1.to_parquet(path=tmpdir, engine="pyarrow")
+    ddf1.to_parquet(path=tmpdir, engine="pyarrow", schema={"col1": pa.decimal128(5, 2)})
     ddf2 = dd.read_parquet(tmpdir, engine="pyarrow")
 
     assert ddf1["col1"].dtype == ddf2["col1"].dtype