Repository: dask/dask

Issue:
Output Parquet filenames with uuid to reduce probability of name collisions
[This issue](https://github.com/dask/dask/issues/6867) discusses a related request.

Suppose you write out files like this:

```python
df.to_parquet('../../tmp/people_compressed_parquet', write_index=False, compression='snappy')
```

Here's the files that'll be outputted:

```
part.0.parquet
part.1.parquet
```

If you write again to the same dir, those files will probably get clobbered.

PySpark outputs filenames like the ones below to sidestep the clobbering issue:

```
part-00000-d13891cd-9c59-4441-a078-cd13d0ebbc1d-c000.snappy.parquet
part-00001-d13891cd-9c59-4441-a078-cd13d0ebbc1d-c000.snappy.parquet
```

The `d13891cd-9c59-4441-a078-cd13d0ebbc1d` part is the random transaction identifier to prevent name collisions.

Adding a full uuid might be a bit excessive.  Perhaps we can output 8 random characters?  I like how Spark adds the compression algo too.  Something like this:

```
part-00000-abcdefgh.snappy.parquet
part-00001-abcdefgh.snappy.parquet
```

Let me know if this suggestion sounds alright and I can try to add a PR.  Thanks!


Discussion:
Thanks @MrPowers. cc @dask/io 
> If you write again to the same dir, those files will probably get clobbered.

We do support `append`, the folder should be explicitly cleared before writing if not appending, or the numbering scheme will continue if yes.

The suggestion isn't unreasonable, but I would worry about ending up with mixed datasets. There is no guarantee that the schema of previous leftover files would match the new, and any `_metadata` file might refer only to some set. It seems to me that writing to a temporary directory and *moving* to the final location upon success is the right pattern, but tricky to implement on any filesystem except local.

> I like how Spark adds the compression algo too.

Strongly oppose this - not only is that information not particularly useful (you cannot take a decision based on it), but parquet actually supports different compression per column and different encoding per data page.
I agree with everything Martin just said, but I also think it is reasonable for the user to have the option to explicitly define a string to be used as the name of every file.  If the user want to include a random uuid and compression information, then so be it. 
That's fair enough. fsspec's open_files for writing allows definition of a format string, but this is not currently available via the dask parquet API.
Thanks for the thoughtful responses and I'll try to give some responses.

I don't feel strongly about any of this and know you're all way more familiar with this codebase and the users.  I'm a Python n00b.  I'm just looking for an excuse to contribute to this code repo 😄 

I did some more experimentation over the weekend and found that CSV files are outputted with no extension, so I'll try to add that via a PR to get started contributing to this project.

> would worry about ending up with mixed datasets. There is no guarantee that the schema of previous leftover files would match the new, and any _metadata file might refer only to some set. It seems to me that writing to a temporary directory and moving to the final location upon success is the right pattern, but tricky to implement on any filesystem except local.

Agree with your concerns, have corrupted data lakes with this very error in the past, and will provide detailed suggestions on a sustainable path forward for this in a future issue.

> Continuing numbering with append

Does this work with concurrent appends?

> Strongly oppose this (outputting compression algo) - not only is that information not particularly useful (you cannot take a decision based on it)

Is there a good way to determine the compression algo of a file if the file is named `part.0.parquet` and you don't have any additional info to work off of?  [This answer](https://stackoverflow.com/a/19127748) says you can make an educated guess from the first few bytes of the file.  Sorry if I'm missing something obvious here.

Will check if `fsspec.utils.infer_compression` can handle this.

> parquet actually supports different compression per column and different encoding per data page.

Didn't realize this, thanks for teaching me!
> Is there a good way to determine the compression algo of a file if the file is named part.0.parquet and you don't have any additional info to work off of? This answer says you can make an educated guess from the first few bytes of the file. Sorry if I'm missing something obvious here.

A "snappy-compressed parquet file" does not mean that you have a normal parquet file and then compress it. Instead, certain data parts within the file are compressed. This is how you have the phenomenally useful feature of parquet to apply different compression algorithms to each column, depending on their data type. Thus, no, you cannot guess the compression from magic bytes, and indeed there is no guarantee of there only being one compression in use. The parquet metadata in the file footer holds all these details.
@martindurant - I now understand your opposition to including the compression algo in the outputted files and agree with you - thanks for helping me understand.

Looks like the `to_csv` method provides the flexibility to output a UUID in the filename:

```python
import uuid
batch_id = uuid.uuid4()
df.to_csv(f"./tmp/csv_appender/mything-{batch_id}-*.csv", index=False)
```

I like this interface and the flexibility it gives users.  This lets you build a system with concurrent CSV appends without worrying about any filename collisions.

Would you be OK exposing this same interface to `to_parquet`?

```python
df.to_parquet(f"./tmp/parquet_appender/mything-*.parquet", write_index=False)
```

Let me know if this proposal sounds alright and I can try to submit a PR.
> Would you be OK exposing this same interface to to_parquet?

In principle yes, but actually parquet writes files into a directory, potentially structured by partitioning. So you wouldn't give an exact glob path like this, but a datafile name template.

Resolving patch:
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -464,6 +464,7 @@ def to_parquet(
     compute=True,
     compute_kwargs=None,
     schema=None,
+    name_function=None,
     **kwargs,
 ):
     """Store Dask.dataframe to Parquet files
@@ -529,6 +530,14 @@ def to_parquet(
         output partition. If the partitions produce inconsistent schemas,
         pyarrow will throw an error when writing the shared _metadata file.
         Note that this argument is ignored by the "fastparquet" engine.
+    name_function : callable, default None
+        Function to generate the filename for each output partition.
+        The function should accept an integer (partition index) as input and
+        return a string which will be used as the filename for the corresponding
+        partition. Should preserve the lexicographic order of partitions.
+        If not specified, files will created using the convention
+        ``part.0.parquet``, ``part.1.parquet``, ``part.2.parquet``, ...
+        and so on for each partition in the DataFrame.
     **kwargs :
         Extra options to be passed on to the specific backend.
 
@@ -537,6 +546,28 @@ def to_parquet(
     >>> df = dd.read_csv(...)  # doctest: +SKIP
     >>> df.to_parquet('/path/to/output/', ...)  # doctest: +SKIP
 
+    By default, files will be created in the specified output directory using the
+    convention ``part.0.parquet``, ``part.1.parquet``, ``part.2.parquet``, ... and so on for
+    each partition in the DataFrame. To customize the names of each file, you can use the
+    ``name_function=`` keyword argument. The function passed to ``name_function`` will be
+    used to generate the filename for each partition and should expect a partition's index
+    integer as input and return a string which will be used as the filename for the corresponding
+    partition. Strings produced by ``name_function`` must preserve the order of their respective
+    partition indices.
+
+    For example:
+
+    >>> name_function = lambda x: f"data-{x}.parquet"
+    >>> df.to_parquet('/path/to/output/', name_function=name_function)  # doctest: +SKIP
+
+    will result in the following files being created::
+
+        /path/to/output/
+            ├── data-0.parquet
+            ├── data-1.parquet
+            ├── data-2.parquet
+            └── ...
+
     See Also
     --------
     read_parquet: Read parquet data to dask.dataframe
@@ -662,8 +693,20 @@ def to_parquet(
         **kwargs_pass,
     )
 
+    # check name_function is valid
+    if name_function is not None and not callable(name_function):
+        raise ValueError("``name_function`` must be a callable with one argument.")
+
     # Use i_offset and df.npartitions to define file-name list
-    filenames = ["part.%i.parquet" % (i + i_offset) for i in range(df.npartitions)]
+    filenames = [
+        f"part.{i + i_offset}.parquet"
+        if name_function is None
+        else name_function(i + i_offset)
+        for i in range(df.npartitions)
+    ]
+
+    if name_function is not None and len(set(filenames)) < len(filenames):
+        raise ValueError("``name_function`` must produce unique filenames.")
 
     # Construct IO graph
     dsk = {}


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -3767,3 +3767,111 @@ def test_extra_file(tmpdir, engine):
             dd.read_parquet(
                 tmpdir, engine=engine, dataset={"require_extension": False}
             ).compute()
+
+
+def test_custom_filename(tmpdir, engine):
+    fn = str(tmpdir)
+    pdf = pd.DataFrame(
+        {"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
+    )
+    df = dd.from_pandas(pdf, npartitions=2)
+    df.to_parquet(fn, name_function=lambda x: f"hi-{x}.parquet", engine=engine)
+
+    files = os.listdir(fn)
+    assert "_common_metadata" in files
+    assert "_metadata" in files
+    assert "hi-0.parquet" in files
+    assert "hi-1.parquet" in files
+    assert_eq(df, dd.read_parquet(fn, engine=engine))
+
+
+def test_custom_filename_works_with_pyarrow_when_append_is_true(tmpdir, engine):
+    fn = str(tmpdir)
+    pdf = pd.DataFrame(
+        {"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
+    )
+    df = dd.from_pandas(pdf, npartitions=2)
+    df.to_parquet(fn, name_function=lambda x: f"hi-{x * 2}.parquet", engine=engine)
+
+    pdf = pd.DataFrame(
+        {"num1": [33], "num2": [44]},
+    )
+    df = dd.from_pandas(pdf, npartitions=1)
+    if engine == "fastparquet":
+        pytest.xfail(
+            "fastparquet errors our with IndexError when ``name_function`` is customized "
+            "and append is set to True.  We didn't do a detailed investigation for expediency. "
+            "See this comment for the conversation: https://github.com/dask/dask/pull/7682#issuecomment-845243623"
+        )
+    df.to_parquet(
+        fn,
+        name_function=lambda x: f"hi-{x * 2}.parquet",
+        engine=engine,
+        append=True,
+        ignore_divisions=True,
+    )
+    files = os.listdir(fn)
+    assert "_common_metadata" in files
+    assert "_metadata" in files
+    assert "hi-0.parquet" in files
+    assert "hi-2.parquet" in files
+    assert "hi-4.parquet" in files
+    expected_pdf = pd.DataFrame(
+        {"num1": [1, 2, 3, 4, 33], "num2": [7, 8, 9, 10, 44]},
+    )
+    actual = dd.read_parquet(fn, engine=engine, index=False)
+    assert_eq(actual, expected_pdf, check_index=False)
+
+
+def test_throws_error_if_custom_filename_is_invalid(tmpdir, engine):
+    fn = str(tmpdir)
+    pdf = pd.DataFrame(
+        {"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
+    )
+    df = dd.from_pandas(pdf, npartitions=2)
+    with pytest.raises(
+        ValueError, match="``name_function`` must be a callable with one argument."
+    ):
+        df.to_parquet(fn, name_function="whatever.parquet", engine=engine)
+
+    with pytest.raises(
+        ValueError, match="``name_function`` must produce unique filenames."
+    ):
+        df.to_parquet(fn, name_function=lambda x: "whatever.parquet", engine=engine)
+
+
+def test_custom_filename_with_partition(tmpdir, engine):
+    fn = str(tmpdir)
+    pdf = pd.DataFrame(
+        {
+            "first_name": ["frank", "li", "marcela", "luis"],
+            "country": ["canada", "china", "venezuela", "venezuela"],
+        },
+    )
+    df = dd.from_pandas(pdf, npartitions=4)
+    df.to_parquet(
+        fn,
+        partition_on=["country"],
+        name_function=lambda x: f"{x}-cool.parquet",
+        write_index=False,
+    )
+
+    for _, dirs, files in os.walk(fn):
+        for dir in dirs:
+            assert dir in (
+                "country=canada",
+                "country=china",
+                "country=venezuela",
+            )
+        for file in files:
+            assert file in (
+                "0-cool.parquet",
+                "1-cool.parquet",
+                "2-cool.parquet",
+                "_common_metadata",
+                "_metadata",
+            )
+    actual = dd.read_parquet(fn, engine=engine, index=False)
+    assert_eq(
+        pdf, actual, check_index=False, check_dtype=False, check_categorical=False
+    )