Repository: yt-project/yt

Issue:
load particles from generic AMReX dataset
I am attempting to read in particle data generated by AMReX and note that if I comment out `AMReXDataset._is_valid()` then I can access my particle data. Essentially I just want the `AMReXDataset._parse_parameter_file()` function, which is effectively permanently disabled by `AMReXDataset._is_valid` always returning `False`. Is it possible to do away with this function and just rely on the `BoxlibDataset` version?

https://github.com/yt-project/yt/blob/4acdd33ac2b9fc646388c24aaf74857aa3bf844c/yt/frontends/boxlib/data_structures.py#L1722


Discussion:
Hi, and welcome to yt!  Thanks for opening your first issue.  We have an issue template that helps us to gather relevant information to help diagnosing and fixing the issue.

Hi @darylbond .
If this method hard-returns False it signals that the class isn't meant to be instantiated by `yt.load()`.
However the class isn't actually used anywhere in the code base, so I think that what's really happening here is that the `_is_valid()` method was never implemented.


> note that if I comment out AMReXDataset._is_valid()

I note that his workaround won't work on the latest version of master since this method is now required in children classes of `Dataset` to be instantiated.

ping @zingale and @matthewturk for possible implementations of this missing bit ?
For reference, what class is your dataset loaded with (if you _don't_ modify the AMRexDataset class) ?
Here's how to inspect it
```python
ds = yt.load(yourfile)
print(type(ds))
```
Actually it looks to me like the file could be validated as long as
1) it's Boxlib valid
2) particles can be found the way `AMRexDataset.parse_parameter_file()` expects them

I could try out this implementation if you are able to share the dataset :)
My apologies, I should have checked I was running the latest version before posting.

The unmodified class loads the dataset as a `yt.frontends.boxlib.data_structures.BoxlibDataset`.

If you could implement this functionality would be greatly appreciated. I was able to access both particle datasets in the attached amrex plotfile using the hack I implemented on version 3.4.0.

[plt.Cavity00010.zip](https://github.com/yt-project/yt/files/5020330/plt.Cavity00010.zip)

> My apologies, I should have checked I was running the latest version before posting.

nothing to apologise for, I was merely pointing out a change for reference !

It worked like a charm, I opened a PR with the fix (#2806). If this doesn't break any existing test it should be quickly available on master.
Feel free to tryout the attached branch to check it works for you too !
> My apologies, I should have checked I was running the latest version before posting.

Reading myself back I realise that my first message may have seemed somewhat passive-aggressive. So sorry about that, it was absolutely not the intended tone :-)
All is well that ends well! I have tested your fix and it works beautifully. Many thanks for the fast turnaround from my posting the issue to having a PR raised, greatly appreciated.
You're welcome. Thanks for reporting and narrowing it down upstream !
Also note that a workaround on the user side that works with any version of yt that has the `AMReXDataset` class is to load directely with
```python
from yt.frontends.boxlib.data_structures import AMReXDataset
ds = AMReXDataset(output_dir, *args, **kwargs)
```
Quick question @darylbond , do you know if the file `inputs` is _always_ named as such for AMRex data ?
an inputs file can have any name.  It is passed via the command line.  Most of the time it will have the word `inputs` in it, but I'm sure there are exceptions.
> an inputs file can have any name. It is passed via the command line. Most of the time it will have the word inputs in it, but I'm sure there are exceptions.

Thank you so much for this, I wasn't aware (and it shows in how I redesigned the code). Fortunately, the current version shows a weakness with OrionDataset _and_ the original validator for this class has exactly the mechanism we need to introduce the required flexibility. I'll patch this in a few minutes
@darylbond 
Here's a quick follow up.

FYI , the _preferred_ way to import the AMReXDataset class should actually be this one
```python
from yt.frontends.boxlib.api import AMReXDataset
```
unfortunately the entries are currently missing in the api module (this is fixed in #2814), so it will work in the future but not with past versions.

Also, while digging into the original fix for this issue I realised there are actually many hidden issues with Boxlib dataset classes in general so I'll need to work a bit more on it before #2806 can be merged.
For that purpose, it would be helpful if we could use your dataset in tests. Would you allow we add it to yt's publicly available sample collection so we can use it to bullet proof the code base in the future ? Thanks !

Thanks for the update.

I have cleaned out some custom entries in the dataset which may have introduced some confusion and reuploaded it here. You are very welcome to use it as a sample.

[plt.amrex.zip](https://github.com/yt-project/yt/files/5032483/plt.amrex.zip)

Thanks a lot for doing this ! I appreciate it :-)
Actually I'm seeing you removed the `inputs` file entirely. It is my understanding that such a file is in fact always to be expected in a BoxLib data directory, and I think it would be better to keep it attached: yt actively uses it to read the simulation parameters. Is it okay for you if I keep it included in the sample ?
I went ahead and uploaded the original full-dataset, but I'm happy to revert this if needed !

edit: for clarity, it's not stored on the data hub yet, I just opened the PR linked above this message.
just catching up here.  There is not requirement that an inputs file is always expected in the BoxLib data directory.  In fact, I am not aware of any codes that do this.

Some codes have adopted some form of a job_info file that has a lot of information about the build environment, machine run, CPU time usage, git hashes, runtime parameter values, etc.  
Okay so if I'm reading the original code right, this means that all BoxLibDatase subclasses have very brittle validators indeed.
#2807 makes this more apparent and easier to fix afterwards. @zingale could you review it ? :)

Resolving patch:
diff --git a/yt/frontends/boxlib/data_structures.py b/yt/frontends/boxlib/data_structures.py
--- a/yt/frontends/boxlib/data_structures.py
+++ b/yt/frontends/boxlib/data_structures.py
@@ -1,5 +1,4 @@
 import glob
-import inspect
 import os
 import re
 import warnings
@@ -634,13 +633,13 @@ class BoxlibDataset(Dataset):
     _index_class = BoxlibHierarchy
     _field_info_class = BoxlibFieldInfo
     _output_prefix = None
-
+    _default_cparam_filename = "job_info"
     periodicity = (False, False, False)
 
     def __init__(
         self,
         output_dir,
-        cparam_filename="job_info",  # todo: harmonise this default value with docstring
+        cparam_filename=None,
         fparam_filename=None,
         dataset_type="boxlib_native",
         storage_filename=None,
@@ -656,6 +655,8 @@ def __init__(
         """
         self.fluid_types += ("boxlib",)
         self.output_dir = os.path.abspath(os.path.expanduser(output_dir))
+
+        cparam_filename = cparam_filename or self.__class__._default_cparam_filename
         self.cparam_filename = self._lookup_cparam_filepath(
             output_dir, cparam_filename=cparam_filename
         )
@@ -688,20 +689,32 @@ def _localize_check(self, fn):
         return None
 
     @classmethod
-    def _is_valid(cls, *args, **kwargs):
-        # fill our args
+    def _is_valid(cls, *args, cparam_filename=None, **kwargs):
         output_dir = args[0]
         header_filename = os.path.join(output_dir, "Header")
         # boxlib datasets are always directories, and
         # We *know* it's not boxlib if Header doesn't exist.
-        return os.path.exists(header_filename)
+        if not os.path.exists(header_filename):
+            return False
+
+        if cls is BoxlibDataset:
+            # Stop checks here for the boxlib base class.
+            # Further checks are performed on subclasses.
+            return True
+
+        cparam_filename = cparam_filename or cls._default_cparam_filename
+        cparam_filepath = cls._lookup_cparam_filepath(output_dir, cparam_filename)
+
+        if cparam_filepath is None:
+            return False
+
+        lines = [line.lower() for line in open(cparam_filepath).readlines()]
+        return any(cls._subtype_keyword in line for line in lines)
 
     @classmethod
-    def _lookup_cparam_filepath(cls, *args, **kwargs):
-        output_dir = args[0]
-        iargs = inspect.getcallargs(cls.__init__, args, kwargs)
+    def _lookup_cparam_filepath(cls, output_dir, cparam_filename):
         lookup_table = [
-            os.path.abspath(os.path.join(p, iargs["cparam_filename"]))
+            os.path.abspath(os.path.join(p, cparam_filename))
             for p in (output_dir, os.path.dirname(output_dir))
         ]
         found = [os.path.exists(file) for file in lookup_table]
@@ -711,21 +724,6 @@ def _lookup_cparam_filepath(cls, *args, **kwargs):
 
         return lookup_table[found.index(True)]
 
-    @classmethod
-    def _is_valid_subtype(cls, *args, **kwargs):
-        # this is used by derived classes
-        output_dir = args[0]
-
-        if not BoxlibDataset._is_valid(output_dir):
-            return False
-
-        cparam_filepath = cls._lookup_cparam_filepath(*args, **kwargs)
-        if cparam_filepath is None:
-            return False
-
-        lines = [line.lower() for line in open(cparam_filepath).readlines()]
-        return any(cls._subtype_keyword in line for line in lines)
-
     def _parse_parameter_file(self):
         """
         Parses the parameter file and establishes the various
@@ -772,8 +770,8 @@ def _parse_cparams(self):
             else:
                 try:
                     vals = _guess_pcast(vals)
-                except IndexError:
-                    # hitting an empty string
+                except (IndexError, ValueError):
+                    # hitting an empty string or a comment
                     vals = None
             self.parameters[param] = vals
 
@@ -1028,11 +1026,12 @@ class OrionDataset(BoxlibDataset):
 
     _index_class = OrionHierarchy
     _subtype_keyword = "hyp."
+    _default_cparam_filename = "inputs"
 
     def __init__(
         self,
         output_dir,
-        cparam_filename="inputs",
+        cparam_filename=None,
         fparam_filename="probin",
         dataset_type="orion_native",
         storage_filename=None,
@@ -1050,10 +1049,6 @@ def __init__(
             unit_system=unit_system,
         )
 
-    @classmethod
-    def _is_valid(cls, *args, **kwargs):
-        return cls._is_valid_subtype(*args, **kwargs)
-
 
 class CastroHierarchy(BoxlibHierarchy):
     def __init__(self, ds, dataset_type="castro_native"):
@@ -1083,11 +1078,12 @@ class CastroDataset(BoxlibDataset):
     _index_class = CastroHierarchy
     _field_info_class = CastroFieldInfo
     _subtype_keyword = "castro"
+    _default_cparam_filename = "job_info"
 
     def __init__(
         self,
         output_dir,
-        cparam_filename="job_info",
+        cparam_filename=None,
         fparam_filename=None,
         dataset_type="boxlib_native",
         storage_filename=None,
@@ -1105,10 +1101,6 @@ def __init__(
             unit_system,
         )
 
-    @classmethod
-    def _is_valid(cls, *args, **kwargs):
-        return cls._is_valid_subtype(*args, **kwargs)
-
     def _parse_parameter_file(self):
         super(CastroDataset, self)._parse_parameter_file()
         jobinfo_filename = os.path.join(self.output_dir, self.cparam_filename)
@@ -1164,11 +1156,12 @@ class MaestroDataset(BoxlibDataset):
 
     _field_info_class = MaestroFieldInfo
     _subtype_keyword = "maestro"
+    _default_cparam_filename = "job_info"
 
     def __init__(
         self,
         output_dir,
-        cparam_filename="job_info",
+        cparam_filename=None,
         fparam_filename=None,
         dataset_type="boxlib_native",
         storage_filename=None,
@@ -1186,10 +1179,6 @@ def __init__(
             unit_system,
         )
 
-    @classmethod
-    def _is_valid(cls, *args, **kwargs):
-        return cls._is_valid_subtype(*args, **kwargs)
-
     def _parse_parameter_file(self):
         super(MaestroDataset, self)._parse_parameter_file()
         jobinfo_filename = os.path.join(self.output_dir, self.cparam_filename)
@@ -1257,11 +1246,12 @@ class NyxDataset(BoxlibDataset):
     _index_class = NyxHierarchy
     _field_info_class = NyxFieldInfo
     _subtype_keyword = "nyx"
+    _default_cparam_filename = "job_info"
 
     def __init__(
         self,
         output_dir,
-        cparam_filename="job_info",
+        cparam_filename=None,
         fparam_filename=None,
         dataset_type="boxlib_native",
         storage_filename=None,
@@ -1279,10 +1269,6 @@ def __init__(
             unit_system,
         )
 
-    @classmethod
-    def _is_valid(cls, *args, **kwargs):
-        return cls._is_valid_subtype(*args, **kwargs)
-
     def _parse_parameter_file(self):
         super(NyxDataset, self)._parse_parameter_file()
 
@@ -1534,11 +1520,12 @@ class WarpXDataset(BoxlibDataset):
     _index_class = WarpXHierarchy
     _field_info_class = WarpXFieldInfo
     _subtype_keyword = "warpx"
+    _default_cparam_filename = "warpx_job_info"
 
     def __init__(
         self,
         output_dir,
-        cparam_filename="warpx_job_info",
+        cparam_filename=None,
         fparam_filename=None,
         dataset_type="boxlib_native",
         storage_filename=None,
@@ -1560,10 +1547,6 @@ def __init__(
             unit_system,
         )
 
-    @classmethod
-    def _is_valid(cls, *args, **kwargs):
-        return cls._is_valid_subtype(*args, **kwargs)
-
     def _parse_parameter_file(self):
         super(WarpXDataset, self)._parse_parameter_file()
         jobinfo_filename = os.path.join(self.output_dir, self.cparam_filename)
@@ -1619,11 +1602,13 @@ def __init__(self, ds, dataset_type="boxlib_native"):
 class AMReXDataset(BoxlibDataset):
 
     _index_class = AMReXHierarchy
+    _subtype_keyword = "amrex"
+    _default_cparam_filename = "job_info"
 
     def __init__(
         self,
         output_dir,
-        cparam_filename="job_info",
+        cparam_filename=None,
         fparam_filename=None,
         dataset_type="boxlib_native",
         storage_filename=None,
@@ -1649,7 +1634,3 @@ def _parse_parameter_file(self):
             self.parameters["particles"] = 1
             self.particle_types = tuple(particle_types)
             self.particle_types_raw = self.particle_types
-
-    @classmethod
-    def _is_valid(cls, *args, **kwargs):
-        return False


Test patch:
diff --git a/yt/frontends/boxlib/tests/test_outputs.py b/yt/frontends/boxlib/tests/test_outputs.py
--- a/yt/frontends/boxlib/tests/test_outputs.py
+++ b/yt/frontends/boxlib/tests/test_outputs.py
@@ -1,6 +1,7 @@
 import numpy as np
 
 from yt.frontends.boxlib.api import (
+    AMReXDataset,
     CastroDataset,
     MaestroDataset,
     NyxDataset,
@@ -304,6 +305,12 @@ def test_WarpXDataset_2():
     assert isinstance(data_dir_load(laser), WarpXDataset)
 
 
+@requires_file("plt.Cavity00010")
+def test_AMReXDataset():
+    ds = data_dir_load("plt.Cavity00010", kwargs={"cparam_filename": "inputs"})
+    assert isinstance(ds, AMReXDataset)
+
+
 @requires_file(rt)
 def test_units_override():
     units_override_check(rt)