Repository: dask/dask

Issue:
Read/write to same file using  `.to_parquet(overwrite=True)` in the same task graph should raise an error
Performing the operation `.to_parquet` produces an error if the Dask dataframe is loaded from a Dask file vs. if it is created within the code and not read from file.

**What happened**: **Way 1** of the code below works as expected - it correctly saves the output to parquet. **Way 2** of the below code, does NOT work as expected and fails with `FileNotFoundError: [Errno 2] No such file or directory: 'error.parquet/part.1.parquet'`

**What you expected to happen**: I expected **Way 2** to work as **Way 1**

**Minimal Complete Verifiable Example**:

```python
import numpy as np
import pandas as pd
import dask.dataframe as dd

# -------------
# way 1 - works
# -------------
print('way 1 - start')
A = np.random.rand(200,300)
cols = np.arange(0, A.shape[1])
cols = [str(col) for col in cols]
df = pd.DataFrame(A, columns=cols)
ddf = dd.from_pandas(df, npartitions=11)

# compute and resave
ddf.drop(cols[0:11], axis=1)
dd.to_parquet(
	ddf, 'error.parquet', engine='auto', compression='default',
	write_index=True, overwrite=True, append=False)
print('way 1 - end')

# ----------------------
# way 2 - does NOT work 
# ----------------------
print('way 2 - start')
ddf = dd.read_parquet('error.parquet')

# compute and resave
ddf.drop(cols[0:11], axis=1)
dd.to_parquet(
	ddf, 'error.parquet', engine='auto', compression='default',
	write_index=True, overwrite=True, append=False)
print('way 2 - end')
```


**Environment**:

- Dask version: dask==2021.3.0
- Python version: 3.8
- Operating System: Mac OS Big Sur 11.2.1
- Install method (conda, pip, source): pip



Discussion:
I provided an answer here: https://stackoverflow.com/questions/66794328/saving-to-parquet-throws-an-error-in-dask-dataframe/66794448#66794448

tl;dr: when overwrite=True, the target path is deleted, which is a problem if dask dataframe is lazy and hopes to read from that path (which is gone by the time it tries to do so)

possible solution: if overwrite=True, then create a temporary folder, save into it and then delete the target folder and replace it with the temporary folder.

problem: this solution can be space- and time-intensive for very large datasets...

Happy to make a PR, but a decision is needed on how to handle the temporary folder.
Great, thanks - that helped. It would be good to have a workaround or fix for this.
BTW, if instead of using 
```python
ddf = ddf.persist()
```

I use 
```python
npartitions = ddf.npartitions
ddf = ddf.compute()
ddf = dd.from_pandas(ddf, npartitions=npartitions)
```

it seems to be faster.
There are many ways to skin a dask dataframe...

On Thursday, March 25, 2021, mengaldo ***@***.***> wrote:

> BTW, instead of using
>
> ddf = ddf.persist()
>
> I use
>
> npartitions = ddf.npartitionsddf = ddf.compute()ddf = dd.from_pandas(ddf, npartitions=npartitions)
>
> it seems to be faster.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/7466#issuecomment-806550843>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE2FWEXBQRXCBVTPRGIMRCLTFMIXJANCNFSM4ZYZLPXQ>
> .
>

Another solution would be to raise an error when a user adds a task that will delete a file that is referenced by a different task in the graph. I wonder if @rjzamora or @crusaderky have ideas about what the behavior should be here.
>Another solution would be to raise an error when a user adds a task that will delete a file that is referenced by a different task in the graph. I wonder if @rjzamora or @crusaderky have ideas about what the behavior should be here.

One of our initial concerns about adding an `overwrite=` argument was that it should really be the users resoponsibility to deal with file/directory cleanup.  We wanted to avoid the need to deal with issues exactly like this one :).  I do think it makes sense to add an explicit check for a circular dependency, but we should avoid adding any special handling (temporary directory creation, etc) within Dask.  If the user wants to read from a dataset and then overwrite it within the same graph execution, **they should probably get an error**.   It should be up to the user to decide how to manage resources (i.e. create a temporary directory, or persist the collection in memory before overwriting the data).

>BTW, if instead of using ... I use ...
>it seems to be faster.

That's strange.
Thanks @jsignell @rjzamora 
I think that having `overwrite=` is a nice feature. Many deployments that we have use a combination of spark and dask, and having some consistency on how the two handle overwriting is good. I think that this issue can be solved, but given my short experience, you are much better placed than me to answer how
Ok like @rjzamora said I think the solution to this issue is:

> If the user wants to read from a dataset and then overwrite it within the same graph execution, **they should probably get an error**

I am updating the title of this issue to reflect that decision.
By changing the way you suggested, it means that 

```python
ddf = ddf.persist()
```
and
```python
npartitions = ddf.npartitions
ddf = ddf.compute()
ddf = dd.from_pandas(ddf, npartitions=npartitions)
```

would still be a workaround to overwrite, correct?

A simple case example in our pipeline is when updating data in a folder. 
Say that you have `./data/filename.parquet`
What we typically do is:
- **read** `ddf = dd.read_parquet('./data/filename.parquet')`
- **update** e.g. `ddf = ddf.merge(ddf_tmp, ...) `
- **resave** `dd.to_parquet(ddf, './data/filename.parquet', write_index=True, overwrite=True, append=False)`

I think this is a fairly common pipeline for real-time operations.

> would still be a workaround to overwrite, correct?

Yes, but keep in mind that if the process crashes for any reason before the writing is complete, you will lose all of the data since it was in memory...
Yes, there is backup; thank you for confirming.
```python
> # ----------------------
> # way 2 - does NOT work 
> # ----------------------
> print('way 2 - start')
> ddf = dd.read_parquet('error.parquet')
> 
> # compute and resave
> ddf.drop(cols[0:11], axis=1)
> dd.to_parquet(
> 	ddf, 'error.parquet', engine='auto', compression='default',
> 	write_index=True, overwrite=True, append=False)
> print('way 2 - end')
>
```

This is IMHO a common dask antipattern. You should put yourself in a position to let dask re-run either any one node of the graph or the whole graph without warning. This is regardless of eagerness or laziness of the implementation.

I would suggest that you write to "error.parquet.new" or similar, and then rename and overwrite client-side after the computation is done, e.g. using ``distributed.Future.add_done_callback``
@crusaderky Thank you for the heads up - will try to implement what you suggested. Would you be able to provide a code snippet on how you would practically modify the above? Thank you!
> I wonder if @rjzamora or @crusaderky have ideas about what the behavior should be here.

pseudocode. Didn't actually check how the graph for read_parquet looks like.
```
def to_parquet(self, name):
    for layer_name, layer in self.dask.layers.items():
        if layer.startswith("read-parquet-"):
            for k, v in layer.items():
                if k[0].startswith("read-parquet-"):
                    prev_name = v[1]
                    if prev_name == name:  # TODO add logic for absolute/relative paths?
                        raise ValueError("read and write on the same file within the same graph")
```
> @crusaderky Thank you for the heads up - will try to implement what you suggested. Would you be able to provide a code snippet on how you would practically modify the above? Thank you!

```python
dd.to_parquet(ddf, 'error.parquet.new', ...)
os.remove('error.parquet')
os.rename('error.parquet.new', 'error.parquet')
```
or
```
d = dd.to_parquet(ddf, 'error.parquet.new', compute=False, ...)
fut, = client.compute(d)
def rename(fut):
   os.remove('error.parquet')
   os.rename('error.parquet.new', 'error.parquet')
fut.add_done_callback(rename)
fut.result()
```
@crusaderky  Thank you!
I'd like to pick this up!

Resolving patch:
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -18,6 +18,7 @@
 from dask.dataframe.methods import concat
 from dask.delayed import Delayed
 from dask.highlevelgraph import HighLevelGraph
+from dask.layers import DataFrameIOLayer
 from dask.utils import apply, import_required, natural_sort_key, parse_bytes
 
 __all__ = ("read_parquet", "to_parquet")
@@ -756,19 +757,34 @@ def to_parquet(
     path = fs._strip_protocol(path)
 
     if overwrite:
-        if _is_local_fs(fs):
-            working_dir = fs.expand_path(".")[0]
-            if path.rstrip("/") == working_dir.rstrip("/"):
-                raise ValueError(
-                    "Cannot clear the contents of the current working directory!"
-                )
         if append:
             raise ValueError("Cannot use both `overwrite=True` and `append=True`!")
+
         if fs.exists(path) and fs.isdir(path):
-            # Only remove path contents if
-            # (1) The path exists
-            # (2) The path is a directory
-            # (3) The path is not the current working directory
+            # Check for any previous parquet layers reading from a file in the
+            # output directory, since deleting those files now would result in
+            # errors or incorrect results.
+            for layer_name, layer in df.dask.layers.items():
+                if layer_name.startswith("read-parquet-") and isinstance(
+                    layer, DataFrameIOLayer
+                ):
+                    path_with_slash = path.rstrip("/") + "/"  # ensure trailing slash
+                    for input in layer.inputs:
+                        if input["piece"][0].startswith(path_with_slash):
+                            raise ValueError(
+                                "Reading and writing to the same parquet file within the "
+                                "same task graph is not supported."
+                            )
+
+            # Don't remove the directory if it's the current working directory
+            if _is_local_fs(fs):
+                working_dir = fs.expand_path(".")[0]
+                if path.rstrip("/") == working_dir.rstrip("/"):
+                    raise ValueError(
+                        "Cannot clear the contents of the current working directory!"
+                    )
+
+            # It's safe to clear the output directory
             fs.rm(path, recursive=True)
 
     # Always skip divisions checks if divisions are unknown


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -3593,6 +3593,29 @@ def test_to_parquet_overwrite_raises(tmpdir, engine):
         dd.to_parquet(ddf, tmpdir, engine=engine, append=True, overwrite=True)
 
 
+def test_to_parquet_overwrite_files_from_read_parquet_in_same_call_raises(
+    tmpdir, engine
+):
+
+    subdir = tmpdir.mkdir("subdir")
+
+    dd.from_pandas(pd.DataFrame({"x": range(20)}), npartitions=2).to_parquet(
+        subdir, engine=engine
+    )
+
+    ddf = dd.read_parquet(subdir)
+
+    # Test writing to the same files, as well as a parent directory
+    for target in [subdir, tmpdir]:
+        with pytest.raises(ValueError, match="same parquet file"):
+            ddf.to_parquet(target, overwrite=True)
+
+        ddf2 = ddf.assign(y=ddf.x + 1)
+
+        with pytest.raises(ValueError, match="same parquet file"):
+            ddf2.to_parquet(target, overwrite=True)
+
+
 def test_to_parquet_errors_non_string_column_names(tmpdir, engine):
     df = pd.DataFrame({"x": range(10), 1: range(10)})
     ddf = dd.from_pandas(df, npartitions=2)