Repository: dask/dask

Issue:
to_parquet fails when append=True
**What happened**: Wrote some Parquet files to a directory.  Then tried to write more Parquet files with `append=True` and the code errored out.

**What you expected to happen**: I expected `append=True` to allow the second write to work.

**Minimal Complete Verifiable Example**:

```python
# this part works
df = pd.DataFrame(
    {"nums": [1, 2, 3, 4, 5, 6], "letters": ["a", "b", "c", "d", "e", "f"]}
)
ddf = dd.from_pandas(df, npartitions=2)

ddf.to_parquet("folder-fastparquet", engine="fastparquet")

df = pd.DataFrame(
    {"nums": [7, 8, 9, 10, 11, 12], "letters": ["g", "h", "i", "j", "k", "l"]}
)
ddf = dd.from_pandas(df, npartitions=3)

# this is the line that fails
ddf.to_parquet("folder-fastparquet", engine="fastparquet", append=True)
```

Here's the error message: ValueError: Appended divisions overlapping with previous ones.
Previous: 5 | New: 0

**Anything else we need to know?**: This also fails when PyArrow is used with a slightly different error message: ValueError: Appended divisions overlapping with the previous ones (set ignore_divisions=True to append anyway).
Previous: 5 | New: 0

**Environment**:

- Dask version: 2021.8.1
- Python version: 3.9.7 
- Operating System: macOS Big Sur
- Install method (conda, pip, source): conda


Discussion:
cc: @dask/io 
Thanks for the nice reproducer @MrPowers. I think there are some assumptions we make about the `divisions` for the partitions for the DataFrame we're writing to disk. As you mentioned, setting `ignore_divisions=True` allows your example to work. Do you have a concrete suggestion as to how the current state of things may be improved? 

Also cc @rjzamora for visibility 
This isn't really a bug - by default `dd.from_pandas` results in a dask dataframe with known divisions. We currently assume that if a user is appending to a dataset that has known divisions (e.g. a sorted index column), that they want to enforce that sorting and do a check. The error message suggests setting `ignore_divisions=True`, which is the recommended solution here. I'll push up a quick patch to make the fastparquet error suggest this as well, and better clarify this in the docs.
Thanks @jcrist, sounds like this behavior is intentional and I confirmed the error message gives a nice hint to set `ignore_divisions=True`, so I'm going to close this one.  Thank you!

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -620,24 +620,28 @@ def initialize_write(
                     tail_metadata.row_group(i)
                     for i in range(tail_metadata.num_row_groups)
                 )
+                index_col_i = names.index(division_info["name"])
                 for row_group in row_groups:
-                    for i, name in enumerate(names):
-                        if name != division_info["name"]:
-                            continue
-                        column = row_group.column(i)
-                        if column.statistics:
-                            if not old_end:
-                                old_end = column.statistics.max
-                            else:
-                                old_end = max(old_end, column.statistics.max)
+                    column = row_group.column(index_col_i)
+                    if column.statistics:
+                        if old_end is None:
+                            old_end = column.statistics.max
+                        elif column.statistics.max > old_end:
+                            old_end = column.statistics.max
+                        else:
+                            # Existing column on disk isn't sorted, set
+                            # `old_end = None` to skip check below
+                            old_end = None
                             break
 
                 divisions = division_info["divisions"]
-                if divisions[0] <= old_end:
+                if old_end is not None and divisions[0] <= old_end:
                     raise ValueError(
-                        "Appended divisions overlapping with the previous ones"
-                        " (set ignore_divisions=True to append anyway).\n"
-                        "Previous: {} | New: {}".format(old_end, divisions[0])
+                        "The divisions of the appended dataframe overlap with "
+                        "previously written divisions. If this is desired, set "
+                        "``ignore_divisions=True`` to append anyway.\n"
+                        "- End of last written partition: {old_end}\n"
+                        "- Start of first new partition: {divisions[0]}"
                     )
 
         extra_write_kwargs = {"schema": schema, "index_cols": index_cols}
diff --git a/dask/dataframe/io/parquet/fastparquet.py b/dask/dataframe/io/parquet/fastparquet.py
--- a/dask/dataframe/io/parquet/fastparquet.py
+++ b/dask/dataframe/io/parquet/fastparquet.py
@@ -1179,11 +1179,13 @@ def initialize_write(
                     else None
                 )
                 divisions = division_info["divisions"]
-                if old_end is None or divisions[0] <= old_end:
+                if old_end is not None and divisions[0] <= old_end:
                     raise ValueError(
-                        "Appended divisions overlapping with previous ones."
-                        "\n"
-                        "Previous: {} | New: {}".format(old_end, divisions[0])
+                        "The divisions of the appended dataframe overlap with "
+                        "previously written divisions. If this is desired, set "
+                        "``ignore_divisions=True`` to append anyway.\n"
+                        "- End of last written partition: {old_end}\n"
+                        "- Start of first new partition: {divisions[0]}"
                     )
         else:
             fmd = fastparquet.writer.make_metadata(


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -873,13 +873,36 @@ def test_append_overlapping_divisions(tmpdir, engine, metadata_file, index, offs
     ddf2 = dd.from_pandas(df.set_index(df.index + offset), chunksize=100)
     ddf1.to_parquet(tmp, engine=engine, write_metadata_file=metadata_file)
 
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(ValueError, match="overlap with previously written divisions"):
         ddf2.to_parquet(tmp, engine=engine, append=True)
-    assert "Appended divisions" in str(excinfo.value)
 
     ddf2.to_parquet(tmp, engine=engine, append=True, ignore_divisions=True)
 
 
+def test_append_known_divisions_to_unknown_divisions_works(tmpdir, engine):
+    tmp = str(tmpdir)
+
+    df1 = pd.DataFrame(
+        {"x": np.arange(100), "y": np.arange(100, 200)}, index=np.arange(100, 0, -1)
+    )
+    ddf1 = dd.from_pandas(df1, npartitions=3, sort=False)
+
+    df2 = pd.DataFrame({"x": np.arange(100, 200), "y": np.arange(200, 300)})
+    ddf2 = dd.from_pandas(df2, npartitions=3)
+
+    # fastparquet always loads all metadata when appending, pyarrow only does
+    # if a `_metadata` file exists. If we know the existing divisions aren't
+    # sorted, then we want to skip erroring for overlapping divisions. Setting
+    # `write_metadata_file=True` ensures this test works the same across both
+    # engines.
+    ddf1.to_parquet(tmp, engine=engine, write_metadata_file=True)
+    ddf2.to_parquet(tmp, engine=engine, append=True)
+
+    res = dd.read_parquet(tmp, engine=engine)
+    sol = pd.concat([df1, df2])
+    assert_eq(res, sol)
+
+
 @pytest.mark.parametrize("metadata_file", [False, True])
 def test_append_different_columns(tmpdir, engine, metadata_file):
     """Test raising of error when non equal columns."""