Repository: ESMCI/cime

Issue:
Does the ERR test truly cover short term archiving?
Looking through the ERR test, it's not clear to me that it actually covers the short-term archiving capabilities. It looks like it's essentially the same as the ERS test, except with these lines:

```
            self._case.set_value("DOUT_S", True)
```

and later:

```
            dout_s_root = self._case.get_value("DOUT_S_ROOT")
            rundir = self._case.get_value("RUNDIR")
            logger.info("staging files from archive %s" % dout_s_root)
            for item in glob.glob(os.path.join(dout_s_root, "rest", "*", "*")):
                shutil.copy(item, rundir)
```

In PR #413 the for loop is changed to:

```
for item in glob.glob(os.path.join(dout_s_root, "*", "hist", "*base")):
```

As a thought experiment, I imagined what would happen if you left out the setting of DOUT_S to True. Then the glob wouldn't pick anything up, in either the old code or the new code. But I believe the second run would proceed just fine, as if you were doing an ERS test.

Which is to say: I believe that this test would pass even if setting DOUT_S to True has absolutely no effect, or if the short-term archiver does absolutely nothing.

Big caveat: I haven't tested this at all. I'd be happy to be shown to be wrong here.

Tagging @mvertens @jgfouca @jedwards4b @bertinia 



Discussion:
I believe this could be fixed by having the test explicitly remove all restart files and history files from the run directory, then doing both the old glob (for restart files) and the new glob (for hist/*base files). In that case, the test would fail if the expected files did not exist in dout_s_root.

This sounds like a good idea. However, I am also wondering if implementing
the ERR test with the clone mechanism you have might not be a better longer
timer solution. Thoughts?

On Tue, Aug 16, 2016 at 9:56 PM, Bill Sacks notifications@github.com
wrote:

> I believe this could be fixed by having the test explicitly remove all
> restart files and history files from the run directory, then doing both the
> old glob (for restart files) and the new glob (for hist/*base files). In
> that case, the test would fail if the expected files did not exist in
> dout_s_root.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/ESMCI/cime/issues/416#issuecomment-240307230, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AHlxEwlAPpfOnoMdtZKLcsREqfLO5HHGks5qgoZlgaJpZM4JmEb2
> .

The ERR test also tests the RESUBMIT and the job dependency mechanism.  The clone method would not test these things.

I agree with @jedwards4b . In addition:
1. I think we'd have to change the ERR test to use a branch run rather than continue run... making it even harder (impossible?) to test the RESUBMIT and job dependency mechanism
2. We'd need to introduce a new hook in the clone-based infrastructure that is called between the two runs (that's not a big deal)

It might have been easier to implement ERR using the original, non-clone-based infrastructure I was working on - but with the changes @jgfouca recently introduced, the value added of that version is small, so I wouldn't suggest adding that as an alternative option.

So I'd recommend not trying to fit ERR into this new mechanism. Overall, I think we'll find that most tests fit cleanly into it, but a few do not.

@billsacks : The hist files with the "base" suffix are removed by the st_archive step, so the test would not pass when the "base" vs "rest" comparison was invoked. I confirmed all this when I was debugging #413

@jgfouca : I agree that that shows that IF the short-term archiver removes hist files with the "base" suffix, THEN it moves them to the correct location. But that doesn't verify that the short-term archiver is actually doing anything - e.g., it may be neither removing the hist files with the "base" suffix NOR moving them to the correct location... and I think the test would still pass in that case.

In addition, I believe the change you made in #413 makes it so that there is no longer any verification that the RESTART files are handled by the short-term archiver. Ideally, it seems that this test would verify that both history and restart files are moved by the short-term archiver.

@billsacks . Ok, I see what you mean. Should the ERR test just remove all the stuff in between phase 1 and phase 2 to ensure it's getting things from the archive?

@jgfouca : I haven't thought about it in depth, but that would be my intuition.

And for the record, it seems like the original ERR test was just as powerless, so this is not a new problem :-)

I think a better test that could capture more of the edge cases associate with the archiver would be to generate a known "truth" archive baseline with a complete set of history and restart files which the st_archive test script can cross-check against for correctness. 

The baseline archive should include as many history output streams as possible (i.e. h0, h1, h2...), multi-instances, and interim restart sets. It could be regenerated by the test suite via a command line switch depending on whether answer changes were introduced into the code or not. 

The test validation criteria could include baseline comparisons with file names, sizes, directory layout, cprnc, and exact restarts comparison from one of the interim restart sets from both the baseline archive and the test generated archive. 

Would this provide better test coverage for the archiver scripts?
@bertinia - that does, indeed, sound like much more rigorous coverage of the archiver. However, I think we'll want to strike a balance here: It sounds like the test you're proposing will fail whenever there are significant code changes. We want the basic test functionality to pass regardless of whether there were changes to the component models (although the baseline comparisons would fail in that case). Moreover, if we keep with the design of other tests, the basic test functionality should not require a baseline directory at all - so, e.g., you could run this on a new machine without needing to first generate baselines. I'm open to the archive test working a little differently from other tests if need be, but I think that should only be done if it's truly necessary (which it may be...).
I looked over this test with @bertinia, and we came up with a way we believe will fix it.  At the beginning of the second phase remove all of the history, restart, and rpointer files.  Next copy all of the files from
the st_archive restart directory.  Then copy the history files back, appending .base to the filename.  This will ensure that we archived all of the files needed for a restart, and checks that history files were archived.  This might also catch some issues that @jedwards4b reported in #1006.

Resolving patch:
diff --git a/scripts/lib/CIME/SystemTests/err.py b/scripts/lib/CIME/SystemTests/err.py
--- a/scripts/lib/CIME/SystemTests/err.py
+++ b/scripts/lib/CIME/SystemTests/err.py
@@ -27,15 +27,15 @@ def run_phase(self):
         else:
             dout_s_root = self._case.get_value("DOUT_S_ROOT")
             rundir = self._case.get_value("RUNDIR")
-            logger.info("staging files from archive %s" % dout_s_root)
-            for item in glob.glob(os.path.join(dout_s_root, "*", "hist", "*base")):
-                # NOTE - this loop doesn't do anything because the
-                # _ers_first_phase has already copied the history files in the
-                # rundir adding the .base suffix before the short term archiver runs.
-                # The pattern matching in the env_archive.xml looks for history 
-                # files that match ".nc$" in the rundir in order to copy them
-                # to the DOUT_S_ROOT location so the files with the .base
-                # suffix are never copied. 
+            case = self._case.get_value("CASE")
+            # First remove restart, history and rpointer files from the run directory
+            for item in glob.iglob(os.path.join(rundir, "%s.*"%case)):
+                if not item.endswith("base"):
+                    os.remove(item)
+            for item in glob.iglob(os.path.join(rundir, "rpointer.*")):
+                os.remove(item)
+            # Then replace them from the restart directory
+            for item in glob.iglob(os.path.join(dout_s_root,"rest","*","*")):
                 shutil.copy(item, rundir)
 
             self._ers_second_phase()


Test patch: