Repository: yt-project/yt

Issue:
Saving a PlotWindow object to a non-existent directory with a file prefix no longer works
### Bug report

In the past, it was possible to save a SlicePlot or ProjectionPlot out to a non-existent directory with a custom prefix. but this no longer works.  Now, you must create the directory first before saving there, or else you get a `FileNotFoundError`.  

What *does* work is this.  

```
import yt
ds = yt.testing.fake_amr_ds()
p = yt.SlicePlot(ds, "z", 'Density')
p.save("images/")
```

It works even when `images` does not yet exist, by first creating the `images` directory and then saving the file `AMRGridData_Slice_z_Density.png` into the `images` directory.


What does not work is this:

```
import yt
ds = yt.testing.fake_amr_ds()
p = yt.SlicePlot(ds, "z", 'Density')
p.save("images/test")
```

If you do not have an `images` directory created, it will fail with a `FileNotFoundError`.  Previously, it would create an `images` directory, then output `test_Slice_z_Density.png` to it, similar to the non-prefixed example above.

The error that I now receive is:

```
Traceback (most recent call last):
  File "test_images.py", line 4, in <module>
    p.save("images/test")
  File "/Users/chummels/src/yt/yt/visualization/plot_container.py", line 76, in newfunc
    rv = f(*args, **kwargs)
  File "/Users/chummels/src/yt/yt/visualization/plot_container.py", line 564, in save
    names.append(v.save(name, mpl_kwargs))
  File "/Users/chummels/src/yt/yt/visualization/base_plot_types.py", line 149, in save
    canvas.print_figure(name, **mpl_kwargs)
  File "/Users/chummels/miniconda3/lib/python3.8/site-packages/matplotlib/backend_bases.py", line 2119, in print_figure
    result = print_method(
  File "/Users/chummels/miniconda3/lib/python3.8/site-packages/matplotlib/backends/backend_agg.py", line 535, in print_png
    with cbook.open_file_cm(filename_or_obj, "wb") as fh:
  File "/Users/chummels/miniconda3/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Users/chummels/miniconda3/lib/python3.8/site-packages/matplotlib/cbook/__init__.py", line 418, in open_file_cm
    fh, opened = to_filehandle(path_or_file, mode, True, encoding)
  File "/Users/chummels/miniconda3/lib/python3.8/site-packages/matplotlib/cbook/__init__.py", line 403, in to_filehandle
    fh = open(fname, flag, encoding=encoding)
FileNotFoundError: [Errno 2] No such file or directory: 'images/test_Slice_z_Density.png'
```

I guess maybe this is minor since people should be already create the directories to where they're saving files?  But given that this used to work up until recently, and that it still works when you don't specify a prefix, maybe it's an easy fix and worth doing?

At the very least, if this is going to be left as is, I think we need to document it on the yt4 differences page.  I tried to run a script that used to work and it broke and it took a while to track down what was going wrong.


Discussion:
This looks very much like a side effect of #2878 and #3176 where I attempted to dedupplicate a lot of code blocks that were handling this kind of stuff in very similar yet subtly different ways. I tried to break as little stuff as possible but honestly I'm not surprised something like this is broken now :/
I'm sorry I didn't document or adverstise it better. In my defense by the time it was considered for inclusion I had forgotten about how much it could affect downstream applications.

Anyway I agree this should be considered both a regression and a bug, and it should be fixed. Thank you for reporting !

Thanks for looking into this, @neutrinoceros !  No blame here, just trying to document where things go wrong for me, so hopefully it makes it easier for someone else too. :)
Actually I changed my mind about this. There are two reasons I see why the old behaviour may be deemed undesirable:

- it isn't consistent with matplotlib (`pyplot.savefig` never creates a missing dir, not even a single level)
- it is very hard to guarantee stability and consistency of this behaviour across yt's api and over time, since it would require adding this line
```python
os.makedirs(os.path.dirname(file), exist_ok=True)
```
in many places, and there would always be a chance that we missed one. It's also arguably bad practice to create nested files implicitly, because the fact that the dirs are missing may be a sign that the application is actually faulty, and it's virtually impossible to correct for that from the outside. On the other hand, it's very easy for the user to obtain the behaviour you want *if* they want it (using the same line above), so I believe it should be the user's responsibility.

I'd even argue we shouldn't create the first dir level implicitly either, though I don't think it warrants a *second* behaviour change in yt 4.0.
If you agree with my reasoning, I suggest we close this and open a different issue (for 4.1) regarding my last point.
I disagree. I think that the auto-creation of a directory during save time is a very useful piece of functionality, and even if MPL doesn't do this, I think our user base has come to use it and expect it.  I certainly use it all the time.  

I guess I don't understand the difficulty with ensuring the codeline above is included in PlotWindow moving forward?  There aren't *that* many places we call `save`.

I am certainly willing to be outvoted on this point, that we should remove the existing functionality that auto-creates a directory when one attempts to save a file to it, but I think it's useful and expected functionality.  What do other people think?
> There aren't that many places we call save.

About half a dozen currently, and I don't think they all share any one common ancestor (inheritance wise), which makes them harder to keep consistent whenever one of them change.

While I still think we should keep away from doing this kind of "magic" (creating data/files/dirs without explicit user consent), you're right that this is currently expectable from Yt since it's been with us for a while. What about reintroducing it on purpose, but with a deprecation warning ? (In which case the target removal version could be yt 4.2)

The auto dir generation could be implemented as a helper function, which would naturally give room for a deprecation cycle.
I think we should bring it up for discussion, before making a decision.  @chummels shoot an email to yt-dev, pointing to this issue so folks know it's up for discussion?
One thing to note that @neutrinoceros and I just found is that `plot_container.py`'s definition of `save()`, which will call individual `PlotMPL` objects to `save()`, does in fact call `ensure_dir` ... so it *should* be working, in some use cases.  I'll dig in more as I am able.
Sorry, to be more clear, it will work if you allow it to set the full filename -- as in, if you call `.save("something/")` but maybe you knew that.
So @brittonsmith weighted in on this during today's triage and he prefers the previous behaviour (both cases in the original post should work). As for me, I like consistency more than I care about my previous argument, so I will fix this.
I forgot to deal with this last week end, I vow to fix it by next Sunday

Resolving patch:
diff --git a/yt/visualization/plot_container.py b/yt/visualization/plot_container.py
--- a/yt/visualization/plot_container.py
+++ b/yt/visualization/plot_container.py
@@ -555,15 +555,24 @@ def save(self, name=None, suffix=".png", mpl_kwargs=None):
         names = []
         if mpl_kwargs is None:
             mpl_kwargs = {}
-        if isinstance(name, (tuple, list)):
-            name = os.path.join(*name)
+
         if name is None:
             name = str(self.ds)
+
+        # ///// Magic area. Muggles, keep out !
+        if isinstance(name, (tuple, list)):
+            name = os.path.join(*name)
+
         name = os.path.expanduser(name)
-        if name[-1] == os.sep and not os.path.isdir(name):
-            ensure_dir(name)
-        if os.path.isdir(name) and name != str(self.ds):
-            name = name + (os.sep if name[-1] != os.sep else "") + str(self.ds)
+
+        parent_dir, _, prefix1 = name.replace(os.sep, "/").rpartition("/")
+        parent_dir = parent_dir.replace("/", os.sep)
+
+        if parent_dir and not os.path.isdir(parent_dir):
+            ensure_dir(parent_dir)
+
+        if name.endswith(("/", os.path.sep)):
+            name = os.path.join(name, str(self.ds))
 
         new_name = validate_image_name(name, suffix)
         if new_name == name:


Test patch:
diff --git a/tests/tests.yaml b/tests/tests.yaml
--- a/tests/tests.yaml
+++ b/tests/tests.yaml
@@ -183,6 +183,7 @@ other_tests:
      - '--ignore-files=test_load_sample.py'
      - '--ignore-files=test_field_access_pytest.py'
      - '--ignore-files=test_ambiguous_fields.py'
+     - "--ignore-files=test_save.py"
      - '--exclude-test=yt.frontends.gdf.tests.test_outputs.TestGDF'
   cookbook:
      - 'doc/source/cookbook/tests/test_cookbook.py'
diff --git a/yt/visualization/tests/test_save.py b/yt/visualization/tests/test_save.py
new file mode 100644
--- /dev/null
+++ b/yt/visualization/tests/test_save.py
@@ -0,0 +1,34 @@
+import os
+
+import yt
+
+
+def test_save_to_path(tmp_path):
+    ds = yt.testing.fake_amr_ds()
+    p = yt.SlicePlot(ds, "z", "Density")
+    p.save(f"{tmp_path}/")
+    assert len(list((tmp_path).glob("*.png"))) == 1
+
+
+def test_save_to_missing_path(tmp_path):
+    # the missing layer should be created
+    ds = yt.testing.fake_amr_ds()
+    p = yt.SlicePlot(ds, "z", "Density")
+
+    # using forward slashes should work even on windows !
+    save_path = os.path.join(tmp_path / "out") + "/"
+    p.save(save_path)
+    assert os.path.exists(save_path)
+    assert len(list((tmp_path / "out").glob("*.png"))) == 1
+
+
+def test_save_to_missing_path_with_file_prefix(tmp_path):
+    # see issue
+    # https://github.com/yt-project/yt/issues/3210
+    ds = yt.testing.fake_amr_ds()
+    p = yt.SlicePlot(ds, "z", "Density")
+    p.save(tmp_path.joinpath("out", "saymyname"))
+    assert (tmp_path / "out").exists()
+    output_files = list((tmp_path / "out").glob("*.png"))
+    assert len(output_files) == 1
+    assert output_files[0].stem.startswith("saymyname")  # you're goddamn right