Repository: dask/dask

Issue:
When updating a parquet dataset on S3 using arrow-engine it fails reading the schema (_get_dataset_offset)
When creating a new parquet (hive partitioned) dataset on S3 using pyarrow this runs fine initially; the parts are created, so is the common/metadata.

But then when updating this dataset with additional chunks of data it asserts.

Notice this is a:
- (hive) partitioned dataset
- pyarrow engine selected
- schema explicitly provided (so no infer of schema)

code fragment:

```python
    ddf = dd.from_pandas(pdf, npartitions=1)
    ddf.to_parquet(
        path="s3://<my-bucket>/<my-path>",
        engine="pyarrow",
        compression="snappy",
        write_index=False,
        append=<initially False, then True>,
        overwrite=False,
        ignore_divisions=True,
        partition_on=['a', 'b', 'c'],
        write_metadata_file=True,
        compute=True,
        schema=schema,
    )
```

Callstack to the assert is as follows:

`  File "/opt/miniconda/envs/igx3/lib/python3.8/site-packages/dask/dataframe/core.py", line 4540, in to_parquet
    return to_parquet(self, path, *args, **kwargs)
  File "/opt/miniconda/envs/igx3/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py", line 644, in to_parquet
    meta, schema, i_offset = engine.initialize_write(
  File "/opt/miniconda/envs/igx3/lib/python3.8/site-packages/dask/dataframe/io/parquet/arrow.py", line 830, in initialize_write
    arrow_schema = fmd.schema.to_arrow_schema()
AttributeError: 'NoneType' object has no attribute 'schema'
`

Dask inspects if there are other file-fragments (and there are). Then pyarrow is invoked to read the metadata (in _get_dataset_offset).

`fmd = pq.read_metadata(fs.sep.join([path, "_metadata"]))`

However there seems to be no support for S3 on that pyarrow function: https://github.com/dask/dask/blob/main/dask/dataframe/io/parquet/arrow.py#L751

Metadata is read, schema is accessible and the update can proceed.

I can image pyarrow engine doesn't support this specific setup, what other engines should support this setup?

And also are there any drawbacks writing directly to S3, any alternatives?

**Environment**:

- fsspec==2021.8.1
- s3fs==2021.8.1
- dask==2021.9.1
- pyarrow==5.0.0
- Python version: 3.8.5
- Operating System: Ubuntu
- Install method: pip



Discussion:
Ping @martindurant for ideas on this one. 
I expect that `engine="fastparquet"` handles this fine. On calling the specific method that does not handle filesystems, I have not yet looked in details, but very probably that pyarrow method can take a file-like object as created by fsspec
```
with fs.open(fs.sep.join([path, "_metadata"])) as f:
    fmd = pq.read_metadata(f)
```

Perhaps your specific code path is not hit in our test suite.

cc @rjzamora 

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -748,7 +748,8 @@ def _get_dataset_offset(cls, path, fs, append, ignore_divisions):
                 # No dataset to append to
                 return fmd, i_offset, False
             try:
-                fmd = pq.read_metadata(fs.sep.join([path, "_metadata"]))
+                with fs.open(fs.sep.join([path, "_metadata"]), mode="rb") as fil:
+                    fmd = pq.read_metadata(fil)
             except (IOError, FileNotFoundError):
                 # No _metadata file present - No appending allowed (for now)
                 if not ignore_divisions:


Test patch:
diff --git a/dask/bytes/tests/test_s3.py b/dask/bytes/tests/test_s3.py
--- a/dask/bytes/tests/test_s3.py
+++ b/dask/bytes/tests/test_s3.py
@@ -435,7 +435,6 @@ def test_parquet(s3, engine, s3so, metadata_file):
     dd = pytest.importorskip("dask.dataframe")
     pd = pytest.importorskip("pandas")
     np = pytest.importorskip("numpy")
-    from dask.dataframe._compat import tm
 
     lib = pytest.importorskip(engine)
     lib_version = parse_version(lib.__version__)
@@ -477,7 +476,61 @@ def test_parquet(s3, engine, s3so, metadata_file):
     )
     assert len(df2.divisions) > 1
 
-    tm.assert_frame_equal(data, df2.compute())
+    dd.utils.assert_eq(data, df2)
+
+
+@pytest.mark.parametrize("engine", ["pyarrow", "fastparquet"])
+def test_parquet_append(s3, engine, s3so):
+    pytest.importorskip(engine)
+    dd = pytest.importorskip("dask.dataframe")
+    pd = pytest.importorskip("pandas")
+    np = pytest.importorskip("numpy")
+
+    url = "s3://%s/test.parquet.append" % test_bucket_name
+
+    data = pd.DataFrame(
+        {
+            "i32": np.arange(1000, dtype=np.int32),
+            "i64": np.arange(1000, dtype=np.int64),
+            "f": np.arange(1000, dtype=np.float64),
+            "bhello": np.random.choice(["hello", "you", "people"], size=1000).astype(
+                "O"
+            ),
+        },
+    )
+    df = dd.from_pandas(data, chunksize=500)
+    df.to_parquet(
+        url,
+        engine=engine,
+        storage_options=s3so,
+        write_index=False,
+    )
+    df.to_parquet(
+        url,
+        engine=engine,
+        storage_options=s3so,
+        write_index=False,
+        append=True,
+        ignore_divisions=True,
+    )
+
+    files = [f.split("/")[-1] for f in s3.ls(url)]
+    assert "_common_metadata" in files
+    assert "_metadata" in files
+    assert "part.0.parquet" in files
+
+    df2 = dd.read_parquet(
+        url,
+        index=False,
+        engine=engine,
+        storage_options=s3so,
+    )
+
+    dd.utils.assert_eq(
+        pd.concat([data, data]),
+        df2,
+        check_index=False,
+    )
 
 
 def test_parquet_wstoragepars(s3, s3so):