Repository: modin-project/modin

Issue:
Non-deterministic FileExistsError when writing dataframe to parquet file
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: 20.04.4 LTS (Focal Fossa)
- **Modin version** (`modin.__version__`): 0.14.1
- **Python version**: Python 3.8.10
- **Code we can use to reproduce**:
```python
import numpy as np
import modin.pandas as pd
import traceback
import shutil
import argparse
import ray
import os


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("--repeats", default=100, type=int)
    parser.add_argument("--unique-filenames", action="store_true", default=False)
    args = parser.parse_args()

    ray.init()
    print(ray.cluster_resources())

    n_cols = 8
    n_rows = 1048576
    array = np.zeros((n_rows, n_cols))
    file_path = "file_exists_error/my_parquet_dir"
    shutil.rmtree("file_exists_error")
    os.makedirs("file_exists_error", exist_ok=True)

    df = pd.DataFrame(array, columns=[f"col{i}" for i in range(n_cols)])
    for i in range(args.repeats):
        if args.unique_filenames:
            file_path_ = file_path + str(i)
        else:
            file_path_ = file_path
        if os.path.exists(file_path):
            shutil.rmtree(file_path)

        try:
            df.to_parquet(file_path_, engine="pyarrow")
        except KeyboardInterrupt as e:
            raise e
        except Exception as e:
            print(traceback.format_exc())
            print("Failed after {} iterations".format(i))
            exit()

    print("Success!")

```
### Describe the problem

When writing dataframes files to disk as parquet files modin sometimes (it's non-deterministic) throws a FileExistsError. This doesn't occur when using pandas.

### Source code / logs

Log for the error is below. I use a different filename for each write so I'm sure that the write directory doesn't already exist before.
```
Traceback (most recent call last):
  File "file_exists_error.py", line 37, in <module>
    df.to_parquet(file_path_, engine="pyarrow")
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/pandas/dataframe.py", line 2214, in to_parquet
    return FactoryDispatcher.to_parquet(new_query_compiler, **config, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/dispatching/factories/dispatcher.py", line 304, in to_parquet
    return cls.__factory._to_parquet(*args, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/dispatching/factories/factories.py", line 423, in _to_parquet
    return cls.io_cls.to_parquet(*args, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py", line 311, in to_parquet
    ray.get([part.oid for row in result for part in row])
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/ray/worker.py", line 1809, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(FileExistsError): ray::deploy_ray_func() (pid=1461969, ip=192.168.32.180)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
    result = func(*args)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
    result = func(dataframe, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py", line 296, in func
    os.makedirs(output_path)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'file_exists_error/my_parquet_dir19'

2022-05-27 16:37:03,947	ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1461969, ip=192.168.32.180)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
    result = func(*args)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
    result = func(dataframe, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py", line 296, in func
    os.makedirs(output_path)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'file_exists_error/my_parquet_dir19'
2022-05-27 16:37:03,948	ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1461969, ip=192.168.32.180)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
    result = func(*args)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
    result = func(dataframe, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py", line 296, in func
    os.makedirs(output_path)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'file_exists_error/my_parquet_dir19'
2022-05-27 16:37:03,948	ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1461969, ip=192.168.32.180)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
    result = func(*args)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
    result = func(dataframe, **kwargs)
  File "/home/oscark@ilab.local/PycharmProjects/tig-team-oscar/projects/modin-playground/venv/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py", line 296, in func
    os.makedirs(output_path)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'file_exists_error/my_parquet_dir19'
```

@m4rs-mt


Discussion:

Resolving patch:
diff --git a/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py b/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
--- a/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
+++ b/modin/core/execution/ray/implementations/pandas_on_ray/io/io.py
@@ -277,6 +277,9 @@ def to_parquet(cls, qc, **kwargs):
         if not cls._to_parquet_check_support(kwargs):
             return RayIO.to_parquet(qc, **kwargs)
 
+        output_path = kwargs["path"]
+        os.makedirs(output_path, exist_ok=True)
+
         def func(df, **kw):
             """
             Dump a chunk of rows as parquet, then save them to target maintaining order.
@@ -289,11 +292,8 @@ def func(df, **kw):
                 Arguments to pass to ``pandas.to_parquet(**kwargs)`` plus an extra argument
                 `partition_idx` serving as chunk index to maintain rows order.
             """
-            output_path = kwargs["path"]
             compression = kwargs["compression"]
             partition_idx = kw["partition_idx"]
-            if not os.path.exists(output_path):
-                os.makedirs(output_path)
             kwargs[
                 "path"
             ] = f"{output_path}/part-{partition_idx:04d}.{compression}.parquet"


Test patch: