Repository: ESMCI/cime

Issue:
More flexible implementation of regex in get_all_hist_files
In the presence of a suffix, and ignoring for a moment the regex part `"\d?_?(\d{4})?`, the function looks for string of the form `${model}.${ext}X${suffix}`, where X is any single char, and `${}` is just a way to represent a generic var name here. The regex string reads something like the following:
```
string = model + "\." + ext + "." + suffix + "$"
```
This has a couple of minor issues.

The first is that the last dot should be `\.`, since these files, as created in `hist_utils.py` are of the form `${filename}.${suffix}`, so we want to match a dot symbol. `"."` will instead match _any_ char.

The second issue is more a quality-of-life one. It would be nice if the user could specify, as extension, a minimal pattern, _without_ including stuff like `.*` to match any optional strings. Say, for instance, that my history files all contain the string `hist`, but may contain other complicated expressions. For instance, I could have
```
my_model.hist.AVERAGE.ndays_x1.2010-01-01-00000.nc         <- daily average, with first write on 1/1/2010 at 0:00
my_model.hist.INSTANT.nsteps_x100.2010-01-01-00000.nc    <- instant output every 100 steps, with first write on 1/1/2010 at 0:00
```
Using the current impl, the regex to capture these would be
```
hist\.(AVERAGE|INSTANT)\.n(step|sec|min|hour|day|month|year)s_x\d*\.\d{4}-\d{2}-\d{2}-\d{5}\.nc
```
which, besides being super tedious to write, can be hard to read for those maintaining the file. Besides, the key part that "signals" that this is indeed a hist file is just the "hist" string. Not to mention the fact that, for monthly output, one may at some point decide to remove the day and time-of-day from the string at the back (leaving, in that example, just "2010-01"). If they do so, they would have to modify the regex (something that a common developer may not know, and have a hard time figuring out where to put their hands).

Notice that, with just `hist` as `hist_file_extension`, we would not match the regex above. However, we _could_ match the regex, if it was
```
string = model + "\." + ext + ".*?" + "\." + suffix + "$"
```
that is, allow _any string_ between the extension and the suffix. This would give the model developers some flexibility for what to put in their history files names, without having to modify `hist_file_extensions`. All that their history file names should have would simply be `hist` (in this case), and that would be enough.

**tl;dr**: I propose to modify the regex for hist files as
```
string = model + "\d?_?(\d{4})?\." + ext + "." + suffix + "$"               <- current
string = model + "\d?_?(\d{4})?\." + ext + ".*?" + "\." + suffix + "$"   <- new
```
to a) fix the last dot (which currently matches any char, rather than the dot symbol) and b) allow model devs to specify a small/compact regex in `hist_file_extensions`, without adding _all details_ of a hist file name.


Discussion:
The plan is to implement the two suggestions above, but we're setting a long term goal to refactor and simplify `config_archive.xml`.

I've begun working on the following:
1. Fix the dot in the second regex pattern.
2. Allow a full regex pattern via `hist_file_extensions` that by passes the static pattern. This will be controlled by a new attribute on `hist_file_extensions` as to not break backwards compatibility.
Hi Jason, thanks for working on this. I'm all in favor of simplifying code, whenever possible.

I'm not sure I understand your second point though. Could you elaborate what you mean by "full regex pattern" and "bypassing the static pattern"?
Ah I see, I misunderstood the second change. You were suggesting this to fix both issues.
```
string = model + "\d?_?(\d{4})?\." + ext + "." + suffix + "$"               <- current
string = model + "\d?_?(\d{4})?\." + ext + ".*?" + "\." + suffix + "$"   <- new
```
Yes. Basically, I want the user to be able to do, say,
```
<comp_archive_spec compname="foo" compclass="atm">
  <hist_file_extension>my\.cool\.ext</hist_file_extension>
</comp_archive_spec>
```
and match not only `foo.my.col.ext.nc.base`, but also `foo.my.coo.ext.something.something.nc.base`. That is, the user should be able to specify the minimal regex necessary to safely match an output file. This includes adding `.*?` between the provided extension and the dot-suffix part.
I want to bring your attention to the case.st_archive testing capability that would be useful here:
```
  
  --test-all            Run tests of st_archiver functionality on config_arvchive.xml
  --test-case           Run tests of st_archiver functionality on env_arvchive.xml

```

Thanks, I wasn't aware of that!

Resolving patch:
diff --git a/CIME/XML/archive_base.py b/CIME/XML/archive_base.py
--- a/CIME/XML/archive_base.py
+++ b/CIME/XML/archive_base.py
@@ -115,9 +115,9 @@ def get_all_hist_files(self, casename, model, from_dir, suffix="", ref_case=None
         for ext in extensions:
             if ext.endswith("$") and has_suffix:
                 ext = ext[:-1]
-            string = model + r"\d?_?(\d{4})?\." + ext
+            string = model + r"\d?_?(\d{4})?\." + ext + r".*?"
             if has_suffix:
-                string += "." + suffix + "$"
+                string += r"\." + suffix + "$"
 
             logger.debug("Regex is {}".format(string))
             pfile = re.compile(string)


Test patch:
diff --git a/CIME/tests/test_unit_xml_archive_base.py b/CIME/tests/test_unit_xml_archive_base.py
new file mode 100644
--- /dev/null
+++ b/CIME/tests/test_unit_xml_archive_base.py
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+
+import os
+import io
+import unittest
+import tempfile
+from contextlib import contextmanager
+from pathlib import Path
+from unittest import mock
+
+from CIME.XML.archive_base import ArchiveBase
+
+TEST_CONFIG_ARCHIVE_XML = """<components version="2.0">
+  <comp_archive_spec compname="eam" compclass="atm">
+    <hist_file_extension>unique\.name\.unique</hist_file_extension>
+  </comp_archive_spec>
+</components>"""
+
+
+class TestXMLArchiveBase(unittest.TestCase):
+    @contextmanager
+    def _setup_environment(self, test_files):
+        with tempfile.TemporaryDirectory() as temp_dir:
+            for x in test_files:
+                Path(temp_dir, x).touch()
+
+            yield temp_dir
+
+    def test_get_all_hist_files(self):
+        archiver = ArchiveBase()
+
+        archiver.read_fd(io.StringIO(TEST_CONFIG_ARCHIVE_XML))
+
+        test_files = [
+            "casename.eam.0001-01-01-0000.nc",
+            "casename.eam.unique.name.unique.0001-01-01-0000.nc",
+            "casename.eam.unique.name.unique.some.extra.0001-01-01-0000.nc",
+            "casename.eam.unique.name.unique.0001-01-01-0000.nc.base",
+            "casename.eam.unique.name.unique.some.extra.0001-01-01-0000.nc.base",
+        ]
+
+        with self._setup_environment(test_files) as temp_dir:
+            hist_files = archiver.get_all_hist_files(
+                "casename", "eam", from_dir=temp_dir
+            )
+
+        test_files.sort()
+        hist_files.sort()
+
+        assert len(hist_files) == 4
+
+        # assert all match except first
+        for x, y in zip(test_files[1:], hist_files):
+            assert x == y, f"{x} != {y}"
+
+    def test_get_all_hist_files_with_suffix(self):
+        archiver = ArchiveBase()
+
+        archiver.read_fd(io.StringIO(TEST_CONFIG_ARCHIVE_XML))
+
+        test_files = [
+            "casename.eam.0001-01-01-0000.nc",
+            "casename.eam.unique.name.unique.0001-01-01-0000.nc",
+            "casename.eam.unique.name.unique.some.extra.0001-01-01-0000.nc",
+            "casename.eam.unique.name.unique.0001-01-01-0000.nc.base",
+            "casename.eam.unique.name.unique.some.extra.0001-01-01-0000.nc.base",
+        ]
+
+        with self._setup_environment(test_files) as temp_dir:
+            hist_files = archiver.get_all_hist_files(
+                "casename", "eam", suffix="base", from_dir=temp_dir
+            )
+
+        assert len(hist_files) == 2
+
+        assert test_files[3] in hist_files
+        assert test_files[4] in hist_files