Repository: yt-project/yt

Issue:
RAMSES loader does not recognize "groups" file structure
<!--To help us understand and resolve your issue, please fill out the form to
the best of your ability.-->
<!--You can feel free to delete the sections that do not apply.-->

### Bug report

**Bug summary**

The RAMSES loader does not recognize the "groups" output structure (`output_XXXXX/group_YYYYY/info_XXXXX.txt`) with the version 4.0.1 of the code. This is due to the way the output structure is recognized by the `RAMSESFileSanitizer` class (see below), and cannot be bypassed by (sym)linking all the output files a new "clean" structure (`output_XXXXX/info_XXXXX.txt`) due to the file sanitizer resolving the symlinks.

**Code for reproduction**

<!--A minimum code snippet required to reproduce the bug, also minimizing the
number of dependencies required.-->

<!-- If you need to use a data file to trigger the issue you're having, consider
using one of the datasets from the yt data hub (http://yt-project.org/data). If
your issue cannot be triggered using a public dataset, you can use the yt
curldrop (https://docs.hub.yt/services.html#curldrop) to share data
files. Please include a link to the dataset in the issue if you use the
curldrop.-->

```python
ds = yt.load("/path/to/ramses/data/output_00075/group_00001/info_00075.txt")
```
or
```python
ds = yt.load("/path/to/symlinked/ramses/data/output_00075/info_00075.txt")
```

**Actual outcome**

<!--The output produced by the above code, which may be a screenshot, console
output, etc.-->

![Screenshot of the issue](https://user-images.githubusercontent.com/9862893/152833343-05653c2d-f431-41d9-828c-289b0aa05508.png)


```
In [5]: ds = yt.load("/data84/obelisk/RHD/OUTPUT_DIR/output_00075/group_00001/info_00075.txt")
---------------------------------------------------------------------------
YTUnidentifiedDataType                    Traceback (most recent call last)
Input In [5], in <module>
----> 1 ds = yt.load("/data84/obelisk/RHD/OUTPUT_DIR/output_00075/group_00001/info_00075.txt")

File ~/.conda/envs/astro/lib/python3.10/site-packages/yt/loaders.py:98, in load(fn, *args, **kwargs)
     95 if len(candidates) > 1:
     96     raise YTAmbiguousDataType(fn, candidates)
---> 98 raise YTUnidentifiedDataType(fn, *args, **kwargs)

YTUnidentifiedDataType: Could not determine input format from `'/data84/obelisk/RHD/OUTPUT_DIR/output_00075/group_00001/info_00075.txt'`.
```

**Expected outcome**
`ds` should actually be the loaded dataset. Testing on a smaller simulation without the "groups" structure works.

**Version Information**
  * Operating System: GNU/Linux CentOS 8 stream 
  * Python Version: Python 3.10.2
  * yt version: 4.0.1, through `conda install --channel conda-forge yt`

**More details**
I think the issue is in the `RAMSESFileSanitizer` class, defined in `frontends/ramses/data_structures.py`. The `__init__` method uses the class method `test_with_standard_file` to check that the file structure is "right" by capturing the value of `iout` from the name of the folder (it should be `00075` in my case, but is resolved to `00001` because of the group structure), and then checking (with the static method `check_standard_files`) that there are files called `info_{iout}.txt` and `amr_{iout}.txt`, which fails in my case because of the wrong assumption for `iout`.

I think the symlinks trick I used doesn't work because the file sanitizer uses `Path().resolve()`, which does resolve the symlink and leads me back to the issue.

**Proposed solution**
I can think of mainly two ways of addressing this: either accounting properly for the possibility of groups or having the option to not resolve the symlinks.

The first solution seems a bit cleaner, and would require to change a bit `test_with_standard_file` and maybe `test_with_folder_name`. Currently the code matches the structure of the name with `OUTPUT_DIR_RE = re.compile(r"(output|group)_(\d{5})")`, but only uses the 2nd part to capture `iout`. Presumably, catching the "group" case is straightforward, but I'm not 100% sure what to do after that and how it would propagate to the rest of the code.

The second option is much quicker to implement: not resolving the symlinks actually solves my issue. So (following a suggestion from @neutrinoceros on Slack), adding a boolean flag to `RAMSESDataset` that would then be passed to `RAMSESFileSanitizer` should do the trick.

I will try to come up with a PR with that second option ASAP!


Discussion:
Hi, and welcome to yt!  Thanks for opening your first issue.  We have an issue template that helps us to gather relevant information to help diagnosing and fixing the issue.

Fixed with https://github.com/yt-project/yt/pull/3786
Sorry to prompt this, but I think #3786 doesn't fully fix the issue? In https://github.com/yt-project/yt/pull/3786#issuecomment-1031844306 you suggested to merge only if doesn't close the issue.

In particular, something like
```python
ds = yt.load("/path/to/ramses/data/output_00075/group_00001/info_00075.txt")
```
still fails, even with the fix. If needed I can reopen an issue, but I'm not sure I have a (proper) fix for that. I can try to think of one though!
> In https://github.com/yt-project/yt/pull/3786#issuecomment-1031844306 you suggested to merge only if doesn't close the issue.

That was before we found a better fix, though it seems that I missed some details here and I should indeed not have closed this issue. Let's reopen
> Sorry to prompt this, but I think #3786 doesn't fully fix the issue? In [#3786 (comment)](https://github.com/yt-project/yt/pull/3786#issuecomment-1031844306) you suggested to merge only if doesn't close the issue.
> 
> In particular, something like
> 
> ```python
> ds = yt.load("/path/to/ramses/data/output_00075/group_00001/info_00075.txt")
> ```
> 
> still fails, even with the fix. If needed I can reopen an issue, but I'm not sure I have a (proper) fix for that. I can try to think of one though!

Could you try loading with
```
ds = yt.load("/path/to/ramses/data/output_00075/group_00001")
```
instead?
This also fails, presumably because of the way `test_with_folder_name` checks for the output number, basically here: https://github.com/yt-project/yt/blob/55c281448695bb9fe3b34920e43fd2f2ada2ad29/yt/frontends/ramses/data_structures.py#L114
For a run with "groups", this will invariably yield `00001`, which doesn't quite work. It's virtually the same issue as for `test_with_standard_file`:
https://github.com/yt-project/yt/blob/55c281448695bb9fe3b34920e43fd2f2ada2ad29/yt/frontends/ramses/data_structures.py#L127

Resolving patch:
diff --git a/yt/frontends/ramses/data_structures.py b/yt/frontends/ramses/data_structures.py
--- a/yt/frontends/ramses/data_structures.py
+++ b/yt/frontends/ramses/data_structures.py
@@ -3,6 +3,7 @@
 from collections import defaultdict
 from itertools import product
 from pathlib import Path
+from typing import Optional, Tuple
 
 import numpy as np
 
@@ -56,7 +57,7 @@ def __init__(self, filename):
 
         # Loop on both the functions and the tested paths
         for path, check_fun in product(paths_to_try, check_functions):
-            ok, output_dir, info_fname = check_fun(path)
+            ok, output_dir, group_dir, info_fname = check_fun(path)
             if ok:
                 break
 
@@ -64,13 +65,8 @@ def __init__(self, filename):
         if not ok:
             return
 
-        # Special case when organized in groups
-        if output_dir.name == "group_00001":
-            self.root_folder = output_dir.parent
-            self.group_name = output_dir.name
-        else:
-            self.root_folder = output_dir
-            self.group_name = None
+        self.root_folder = output_dir
+        self.group_name = group_dir.name if group_dir else None
         self.info_fname = info_fname
 
     def validate(self) -> None:
@@ -79,18 +75,19 @@ def validate(self) -> None:
         filename: str = os.path.expanduser(self.original_filename)
 
         if not os.path.exists(filename):
-            raise FileNotFoundError(f"No such file or directory {filename!r}")
+            raise FileNotFoundError(rf"No such file or directory '{filename!s}'")
         if self.root_folder is None:
             raise ValueError(
-                f"Could not determine output directory from {filename!r}\n"
-                f"Expected a directory name of form {OUTPUT_DIR_EXP!r}"
+                f"Could not determine output directory from '{filename!s}'\n"
+                f"Expected a directory name of form {OUTPUT_DIR_EXP!r} "
+                "containing an info_*.txt file and amr_* files."
             )
 
         # This last case is (erroneously ?) marked as unreachable by mypy
         # If/when this bug is fixed upstream, mypy will warn that the unused
         # 'type: ignore' comment can be removed
         if self.info_fname is None:  # type: ignore [unreachable]
-            raise ValueError(f"Failed to detect info file from {filename!r}")
+            raise ValueError(f"Failed to detect info file from '{filename!s}'")
 
     @property
     def is_valid(self) -> bool:
@@ -109,37 +106,70 @@ def check_standard_files(folder, iout):
         ok &= (folder / f"info_{iout}.txt").is_file()
         return ok
 
+    @staticmethod
+    def _match_output_and_group(
+        path: Path,
+    ) -> Tuple[Path, Optional[Path], Optional[str]]:
+
+        # Make sure we work with a directory of the form `output_XXXXX`
+        for p in (path, path.parent):
+            match = OUTPUT_DIR_RE.match(p.name)
+
+            if match:
+                path = p
+                break
+
+        if match is None:
+            return path, None, None
+
+        iout = match.group(1)
+
+        # See whether a folder named `group_YYYYY` exists
+        group_dir = path / "group_00001"
+        if group_dir.is_dir():
+            return path, group_dir, iout
+        else:
+            return path, None, iout
+
     @classmethod
-    def test_with_folder_name(cls, output_dir):
-        iout_match = OUTPUT_DIR_RE.match(output_dir.name)
-        ok = output_dir.is_dir() and iout_match is not None
+    def test_with_folder_name(
+        cls, output_dir: Path
+    ) -> Tuple[bool, Optional[Path], Optional[Path], Optional[Path]]:
+        output_dir, group_dir, iout = cls._match_output_and_group(output_dir)
+        ok = output_dir.is_dir() and iout is not None
+
+        info_fname: Optional[Path]
+
         if ok:
-            iout = iout_match.group(2)
-            ok &= cls.check_standard_files(output_dir, iout)
-            info_fname = output_dir / f"info_{iout}.txt"
+            parent_dir = group_dir or output_dir
+            ok &= cls.check_standard_files(parent_dir, iout)
+            info_fname = parent_dir / f"info_{iout}.txt"
         else:
-            output_dir, info_fname = None, None
+            info_fname = None
 
-        return ok, output_dir, info_fname
+        return ok, output_dir, group_dir, info_fname
 
     @classmethod
-    def test_with_standard_file(cls, filename):
-        iout_match = OUTPUT_DIR_RE.match(filename.parent.name)
+    def test_with_standard_file(
+        cls, filename: Path
+    ) -> Tuple[bool, Optional[Path], Optional[Path], Optional[Path]]:
+        output_dir, group_dir, iout = cls._match_output_and_group(filename.parent)
         ok = (
             filename.is_file()
             and STANDARD_FILE_RE.match(filename.name) is not None
-            and iout_match is not None
+            and iout is not None
         )
 
+        info_fname: Optional[Path]
+
         if ok:
-            iout = iout_match.group(2)
-            ok &= cls.check_standard_files(filename.parent, iout)
-            output_dir = filename.parent
-            info_fname = output_dir / f"info_{iout}.txt"
+            parent_dir = group_dir or output_dir
+            ok &= cls.check_standard_files(parent_dir, iout)
+            info_fname = parent_dir / f"info_{iout}.txt"
         else:
-            output_dir, info_fname = None, None
+            info_fname = None
 
-        return ok, output_dir, info_fname
+        return ok, output_dir, group_dir, info_fname
 
 
 class RAMSESDomainFile:
diff --git a/yt/frontends/ramses/definitions.py b/yt/frontends/ramses/definitions.py
--- a/yt/frontends/ramses/definitions.py
+++ b/yt/frontends/ramses/definitions.py
@@ -53,7 +53,7 @@ def ramses_header(hvals):
 # on the left hand side
 VAR_DESC_RE = re.compile(r"\s*([^\s]+),\s*([^\s]+),\s*([^\s]+)")
 
-OUTPUT_DIR_EXP = r"(output|group)_(\d{5})"
+OUTPUT_DIR_EXP = r"output_(\d{5})"
 OUTPUT_DIR_RE = re.compile(OUTPUT_DIR_EXP)
 STANDARD_FILE_RE = re.compile(r"((amr|hydro|part|grav)_\d{5}\.out\d{5}|info_\d{5}.txt)")
 


Test patch:
diff --git a/tests/tests.yaml b/tests/tests.yaml
--- a/tests/tests.yaml
+++ b/tests/tests.yaml
@@ -196,6 +196,7 @@ other_tests:
      - "--ignore-files=test_load_archive\\.py"
      - "--ignore-files=test_outputs_pytest\\.py"
      - "--ignore-files=test_normal_plot_api\\.py"
+     - "--ignore-file=test_file_sanitizer\\.py"
      - "--exclude-test=yt.frontends.gdf.tests.test_outputs.TestGDF"
      - "--exclude-test=yt.frontends.adaptahop.tests.test_outputs"
      - "--exclude-test=yt.frontends.stream.tests.test_stream_particles.test_stream_non_cartesian_particles"
diff --git a/yt/frontends/ramses/tests/test_file_sanitizer.py b/yt/frontends/ramses/tests/test_file_sanitizer.py
new file mode 100644
--- /dev/null
+++ b/yt/frontends/ramses/tests/test_file_sanitizer.py
@@ -0,0 +1,108 @@
+import re
+import tempfile
+from collections import namedtuple
+from itertools import chain
+from pathlib import Path
+
+import pytest
+
+from yt.frontends.ramses.data_structures import RAMSESFileSanitizer
+
+PathTuple = namedtuple(
+    "PathTuple", ("output_dir", "group_dir_name", "info_file", "paths_to_try")
+)
+
+
+def generate_paths(create):
+    with tempfile.TemporaryDirectory() as tmpdir:
+        output_dir = Path(tmpdir) / "output_00123"
+        output_dir.mkdir()
+        amr_file = output_dir / "amr_00123.out00001"
+        if create:
+            amr_file.touch()
+        info_file = output_dir / "info_00123.txt"
+        if create:
+            info_file.touch()
+
+        # Test with regular structure
+        output_dir2 = Path(tmpdir) / "output_00124"
+        output_dir2.mkdir()
+
+        group_dir2 = output_dir2 / "group_00001"
+        group_dir2.mkdir()
+        info_file2 = group_dir2 / "info_00124.txt"
+        if create:
+            info_file2.touch()
+        amr_file2 = group_dir2 / "amr_00124.out00001"
+        if create:
+            amr_file2.touch()
+
+        yield (
+            PathTuple(
+                output_dir=output_dir,
+                group_dir_name=None,
+                info_file=info_file,
+                paths_to_try=(output_dir, info_file, amr_file),
+            ),
+            PathTuple(
+                output_dir=output_dir2,
+                group_dir_name=group_dir2.name,
+                info_file=info_file2,
+                paths_to_try=(output_dir2, info_file2, group_dir2, amr_file2),
+            ),
+        )
+
+
+@pytest.fixture
+def valid_path_tuples():
+    yield from generate_paths(create=True)
+
+
+@pytest.fixture
+def invalid_path_tuples():
+    yield from generate_paths(create=False)
+
+
+def test_valid_sanitizing(valid_path_tuples):
+    for answer in valid_path_tuples:
+        for path in answer.paths_to_try:
+            sanitizer = RAMSESFileSanitizer(path)
+            sanitizer.validate()
+
+            assert sanitizer.root_folder == answer.output_dir
+            assert sanitizer.group_name == answer.group_dir_name
+            assert sanitizer.info_fname == answer.info_file
+
+
+def test_invalid_sanitizing(valid_path_tuples, invalid_path_tuples):
+    for path in chain(*(pt.paths_to_try for pt in invalid_path_tuples)):
+        sanitizer = RAMSESFileSanitizer(path)
+
+        if path.exists():
+            expected_error = ValueError
+            expected_error_message = (
+                "Could not determine output directory from '.*'\n"
+                "Expected a directory name of form .* "
+                "containing an info_\\*.txt file and amr_\\* files."
+            )
+        else:
+            expected_error = FileNotFoundError
+            expected_error_message = re.escape(
+                f"No such file or directory '{str(path)}'"
+            )
+
+        with pytest.raises(expected_error, match=expected_error_message):
+            sanitizer.validate()
+
+    for path in chain(*(pt.paths_to_try for pt in valid_path_tuples)):
+        expected_error_message = re.escape(
+            f"No such file or directory '{str(path/'does_not_exist.txt')}'"
+        )
+        sanitizer = RAMSESFileSanitizer(path / "does_not_exist.txt")
+        with pytest.raises(FileNotFoundError, match=expected_error_message):
+            sanitizer.validate()
+
+    expected_error_message = "No such file or directory '.*'"
+    sanitizer = RAMSESFileSanitizer(Path("this") / "does" / "not" / "exist")
+    with pytest.raises(FileNotFoundError, match=expected_error_message):
+        sanitizer.validate()