Repository: ESMCI/cime

Issue:
St archive test
Add a test of the st_archive capability.  
After any case is created and setup a user can run case.st_archive --test to test the functionality of 
the short term archiver.   

Test suite: scripts_regression_tests.py including new test of st_archive
Test baseline: 
Test namelist changes: 
Test status: bit for bit
Fixes #1927 
Fixes #1589 
Fixes #2488 

User interface changes?: case.st_archive now has a --test option

Update gh-pages html (Y/N)?:

Code review: 



Discussion:
Thanks for working on this. I'm slammed with a lot of other demands right now, so I'm not going to have time to look at this carefully enough to review it... so removing my name as reviewer.
@billsacks I hope that you can at least add a list of files to test for cism and move the cism section to your own cime_config directory.   
@jedwards4b I'm a little confused on what you want from components here, and when. Do you want people to add to this branch? Or is this just showing the idea and you want people to eventually add things to their own config directory? And in terms of adding to the components' config directories: can that only be done after this PR is on master, or is the infrastructure already in place to support that?
@billsacks - You either need to add to this branch or you can add to your component after this branch is merged.   The infrastructure is already in place to read the file if it's there but not the current format of the file which adds the test information.  
From what we said at the CSEG meeting this morning. The agreed upon process, was to get these changes in place into cime. And then for components to update to the version of cime that uses it, and move the appropriate parts to a components own config_archive.xml afterwards. So right now we just approve the changes here, and later we'll have our own version in each component.
@jedwards4b I'm not sure how you wanted us to incorporate changes to our components into this PR. I've done this by making a separate PR into your branch for your review, adding CISM files (https://github.com/jedwards4b/cime/pull/5).

A few key things this covers (relative to CAM/CLM):

- Copy all hist (not just the last)

- Also copy initial_hist (CISM-specific)

- Only move last restart file

I'm not sure if these all make sense or will work with the infrastructure you've set up.
Opening a PR to the branch makes sense but I'm puzzled by this statement:
> I'm not sure if these all make sense or will work with the infrastructure you've set up.

Did you not test the branch?   Was it not clear how to test?
> Did you not test the branch? Was it not clear how to test?

Yes and yes – I hadn't tested it because it wasn't clear how to test. I've reread the PR and have run `./case.st_archive --test` on a case created with `./create_newcase --case t_st_archive_0417a --compset T1850G --res f09_g16 --run-unsupported`. This gave some output, but it's very unclear to me if it's working right.
If it fails - if any file is handled with a different disposition than indicated you will get a clear fail message.   You can test this by intentionally changing a file disposition to something you expect to fail.
To elaborate: Whether I test with a T compset (CISM only) or an I compset (CLM & CISM): after running case.setup and case.st_archive --test, I get output that looks like this:

```
moving st_archive_test/cpl_0001.log.5548574.chadmin1.180228-124723.gz to st_archive_test/archive/logs/cpl_0001.log.5548574.chadmin1.180228-124723.gz
-------------------------------------------
Archiving restarts for date date(1976, 1, 1, 0, 0, 0)
-------------------------------------------
Archiving restarts for dwav (wav)
Archiving history files for dwav (wav)
Restoring restart from st_archive_test/archive/rest/1976-01-01-00000
Restoring st_archive_test/archive/rest/1976-01-01-00000/rpointer.lnd from st_archive_test/archive/rest/1976-01-01-00000 to st_archive_test
Restoring st_archive_test/archive/rest/1976-01-01-00000/rpointer.atm from st_archive_test/archive/rest/1976-01-01-00000 to st_archive_test
Restoring st_archive_test/archive/rest/1976-01-01-00000/rpointer.drv_0001 from st_archive_test/archive/rest/1976-01-01-00000 to st_archive_test
Restoring st_archive_test/archive/rest/1976-01-01-00000/rpointer.glc from st_archive_test/archive/rest/1976-01-01-00000 to st_archive_test
Restoring st_archive_test/archive/rest/1976-01-01-00000/rpointer.drv from st_archive_test/archive/rest/1976-01-01-00000 to st_archive_test
```

and a directory that looks like this:

```
st_archive_test/
├── archive
│   ├── logs
│   │   └── cpl_0001.log.5548574.chadmin1.180228-124723.gz
│   ├── rest
│   │   └── 1976-01-01-00000
│   │       ├── rpointer.atm
│   │       ├── rpointer.drv
│   │       ├── rpointer.drv_0001
│   │       ├── rpointer.glc
│   │       └── rpointer.lnd
│   └── wav
│       └── hist
├── rpointer.atm
├── rpointer.drv
├── rpointer.drv_0001
├── rpointer.glc
└── rpointer.lnd
```

Am I expected to manually verify that this worked correctly? In any case, I expected to see some of the test hist / restart files.
I'm missing something here: I tried changing all of the file dispositions for CLM:

```diff
diff --git a/config/cesm/config_archive.xml b/config/cesm/config_archive.xml
index e4442e5..8a8f515 100644
--- a/config/cesm/config_archive.xml
+++ b/config/cesm/config_archive.xml
@@ -44,15 +44,15 @@
       <rpointer_content>./$CASE.clm2$NINST_STRING.r.$DATENAME.nc</rpointer_content>
     </rpointer>
     <test_file_names>
-      <tfile disposition="copy">rpointer.lnd</tfile>
-      <tfile disposition="copy">rpointer_9999.lnd</tfile>
-      <tfile disposition="copy">casename.clm2.r.1976-01-01-00000.nc</tfile>
-      <tfile disposition="copy">casename.clm2.rh4.1976-01-01-00000.nc</tfile>
-      <tfile disposition="move">casename.clm2.h0.1976-01-01-00000.nc</tfile>
-      <tfile disposition="ignore">casename.clm2.h0.1976-01-01-00000.nc.base</tfile>
-      <tfile disposition="move">casename.clm2_0002.e.postassim.1976-01-01-00000.nc</tfile>
-      <tfile disposition="move">casename.clm2_0002.e.preassim.1976-01-01-00000.nc</tfile>
-      <tfile disposition="ignore">anothercasename.clm2.i.1976-01-01-00000.nc</tfile>
+      <tfile disposition="move">rpointer.lnd</tfile>
+      <tfile disposition="ignore">rpointer_9999.lnd</tfile>
+      <tfile disposition="move">casename.clm2.r.1976-01-01-00000.nc</tfile>
+      <tfile disposition="ignore">casename.clm2.rh4.1976-01-01-00000.nc</tfile>
+      <tfile disposition="copy">casename.clm2.h0.1976-01-01-00000.nc</tfile>
+      <tfile disposition="copy">casename.clm2.h0.1976-01-01-00000.nc.base</tfile>
+      <tfile disposition="copy">casename.clm2_0002.e.postassim.1976-01-01-00000.nc</tfile>
+      <tfile disposition="ignore">casename.clm2_0002.e.preassim.1976-01-01-00000.nc</tfile>
+      <tfile disposition="copy">anothercasename.clm2.i.1976-01-01-00000.nc</tfile>
     </test_file_names>
   </comp_archive_spec>
```

then recreated an I compset case, did case.setup and case.st_archive --test, but the results were no different; no failures were reported.
Nope - there seems to be a problem, I'm working on it. 
I guess I am still confused. I assume the NNNN extensions are the instances, but why are these 9999, 0001, or 0002? Here is what I would do for the CICE without the instances:

    <test_file_names>
      <tfile disposition="copy">rpointer.ice</tfile>
      <tfile disposition="copy">casename.cice.r.1976-01-01-00000.nc</tfile>
      <tfile disposition="move">casename.cice.h.1976-01-01-00000.nc</tfile>
    </test_file_names>


@billsacks please try again - I think I've addressed the problem.
that's fine

On Tue, Apr 17, 2018 at 1:31 PM, dabail10 <notifications@github.com> wrote:

> I guess I am still confused. I assume the NNNN extensions are the
> instances, but why are these 9999, 0001, or 0002? Here is what I would do
> for the CICE without the instances:
>
> <test_file_names>
>   <tfile disposition="copy">rpointer.ice</tfile>
>   <tfile disposition="copy">casename.cice.r.1976-01-01-00000.nc</tfile>
>   <tfile disposition="move">casename.cice.h.1976-01-01-00000.nc</tfile>
> </test_file_names>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/pull/2470#issuecomment-382113740>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AF16GKPiLZwCWzNs6PD1vTrktYsN26tjks5tpkLzgaJpZM4TT6LO>
> .
>



-- 
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

Erik asked this earlier, but are we supposed to push the changes to your fork/branch?
@dabail10 You can open a PR to my branch.
OK. I did not have permission to push to my own branch though.

Dave
I tried to create a branch on your repo so that I can issue a PR from there. It will not let me push to the branch.

Dave
You would checkout my branch, make your changes, push to your fork, open PR to my branch.
That's what I was missing was pushing to my fork.

Dave
Thanks, @jedwards4b . This is working better now. However, I have found a few problems:

(1) Some of the files listed here are not created, and changing their disposition sometimes doesn't cause the test to fail:

```xml
      <tfile disposition="copy">rpointer_9999.glc</tfile>
      <tfile disposition="ignore">casename.cism.r.1975-01-01-00000.nc</tfile>
      <tfile disposition="ignore">anothercasename.cism.r.1976-01-01-00000.nc</tfile>
```

This should be fixed to create those files. Also, to ensure that problems aren't silently ignored, it would be good if this ran through all listed files and ensured that they exist in the test directory before running the archiver.

(2) Changing this from 'ignore' to 'copy' does NOT cause a failure:

```xml
      <tfile disposition="ignore">casename.cism.h.1976-01-01-00000.nc.base</tfile>
```

I'm thinking this means that the logic for checking 'ignore' isn't strict enough.


I think it's actually the logic for copy that isn't strict enough.   Can you be more specific about what files you think are not created - I do check that.
> I think it's actually the logic for copy that isn't strict enough. Can you be more specific about what files you think are not created - I do check that.

You're right - I mis-stated.

The three files listed under (1) in my above comment did not seem to appear anywhere in my `st_archive_test` directory once the test was done running. Actually, once I think I saw the 'anothercasename' file, but most times it didn't appear. (I was repeatedly running the test from the same case directory, rm -rf'ing the st_archive_test directory between each run.)

Since I won't have time to retest this, here's what I was testing, so that you can redo these tests once you have the fixes in place:

(1) Check whether each file listed in the cism section appeared somewhere in the `st_archive_test` directory after the test was done.

(2) Checked whether all combos were handled correctly:

(a) For one file listed as "ignore": changed to "move" and "copy", made sure each gave an error

(b) For one file listed as "copy": changed to "move" and "ignore", made sure each gave an error

(c) For one file listed as "move": changed to "copy" and "ignore", made sure each gave an error

@billsacks thanks for the testing you've done.   Note that the rpointer file should be rpointer.glc_9999 not rpointer_9999.glc
@jedwards4b I simply followed what was done for atm and lnd. Can you please fix the glc one when you fix the atm and lnd ones?
@billsacks thanks again for the careful testing.  I am now able to run all of your suggested tests successfully.
I would like to rebase this PR to resolve conflict and merge it to master.  Please let me know if you are still working on this branch and would like me to wait. 
Great, thanks @jedwards4b - and thanks for putting together this test!
Yes, thanks! I'll look over it when you've rebased
I've worked through the changes, but haven't tested it yet,
pending a few questions.

In config/cesm/config_archive.xml I can add some file names,
but is it OK if the files are fake?  My impression is that this st_archive
test is not being run within scripts_regression_tests.py, so we don't
need to make that run a CAM+DART case to create actual files.
So fake files should be OK for now.

Is the new "disposition" feature going to be used for non-testing
runs of case.st_archive?

Tangential questions:
I noticed that your case_st_archive.py:_get_datename still accepts 3 arguments,
while I thought that we settled on just passing 'case' to it.  Was it necessary to
change back, or did my change get lost in the shuffle?
Similarly for _archive_restarts_date_comp arguments casename, rundir.

There were several logger messages in which I put in some \n to help with
readability of the output, but those aren't in your version.  Is that intentional?  
If so, what's the guideline that I wasn't following?
@kdraeder you can add file names to the test section of the xml - those file names should have casename 'casename' and date 1976-01-01 

The disposition is only for tests.   You are telling the test what the disposition should be and the test is confirming that.   

I did need to shift a few arguments around to get things to work, I may have also changed some formatting - if there is something you feel strongly should be there I can put it back.
I've updated with a rebase, if you have a local copy you will need to do a hard reset:
git fetch jedwards_fork
git reset --hard jedwards_fork/st_archive_test
I've run a test with a few more file names added to the dart section of config_archive.xml.
It does a lot right, but fails on a file name that was already in there. 
> Checking testfile casename.pop.h.1976-01-01-00000.nc with disposition move
> ERROR: Failed to move file casename.pop.h.1976-01-01-00000.nc to archive

That file exists in 
/glade/p/work/raeder/Models/cesm2_0_alpha08g/cime/scripts/st_arch_test1/st_archive_test

While trying to track that down I was puzzled by the messages in _check_disposition:
```
                if "move" in disposition:
                    if find_files(os.path.join(testdir, "archive"), _file):
                        expect(False,
                               "Copied file {} to archive with disposition move".format(_file))
```

Why does it say 'Copied' in the 'move' section?
Then 
```
            elif "ignore" in disposition:
                expect(False, "Moved file {} with dispostion ignore to directory {}".format(_file, root))
            elif "copy" in disposition:
                expect(_file in copyfilelist, "File {} with disposition copy was moved to directory {}"
                       .format(_file, root))
```
Why does 'ignore' lead to 'Moved'?
Why does disposition 'copy' lead to 'moved' (to a non-testdir directory)?


@kdraeder I don't get any error on casename.pop.h.1976-01-01-00000.nc are you sure you are up to date with the branch?   git pull jedwards st_archive_test 
 Thanks for the explanation of _check_disposition.

I still get the failure after doing
 1554  15:26:36 18 Apr  2018 git pull jedwards4b st_archive_test
> git pull jedwards4b st_archive_test 
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 8 (delta 6), reused 8 (delta 6), pack-reused 0
Unpacking objects: 100% (8/8), done.
From https://github.com/jedwards4b/cime
 * branch            st_archive_test -> FETCH_HEAD
   6fde992..948da2e  st_archive_test -> jedwards4b/st_archive_test
Updating 6fde992..948da2e
Fast-forward
 config/cesm/machines/template.st_archive | 33 +++++++++++++++++++++++++--------
 config/e3sm/machines/template.st_archive | 33 +++++++++++++++------------------
 2 files changed, 40 insertions(+), 26 deletions(-)
> ./case.st_archive --test >> & README.case
(I didn't case.setup or case.build after the pull).
Are you working on cheyenne?  Can you please point me to your case directory.
cheyenne.
/glade/p/work/raeder/Models/cesm2_0_alpha08g/cime/scripts/st_arch_test1

I built it with --multi-driver, which may be one difference from others' tests.

@kdraeder You are correct, the test doesn't work if you run it in a multi-driver case directory.   I've added code so that it will fail if you attempt to do so.   Note that this doesn't mean that it doesn't test multidriver cases, it does, but you need to be in a non-multidriver case for it to work.   I'll open an issue to deal with that problem later.   #2480
@jedwards4b 
My single instance, single driver test works until it _checks
the dart restart history files, e.g. casename.dart.e.cam_preassim_priorinf_mean.1976-01-01-00000.nc
which I've tagged with 'copy' in the <tfile> line. 
The testing doesn't include or generate an actual dart.r file, 
which would provide the names of the files to be copied, instead of moved.  
That should that be part of this testing, but could be a little awkward.
We could create a dart.r file with the test file names in it using something
like the script I provided in #1927.  
How is this handled for CLM restart history files?
Now fixes #2488 
reviewers - we would like to merge this PR this morning so that we can include it in cesm testing - any objections?
@jedwards4b everything looks great to me except for case_st_archive.py which has more changes than I am able to review/understand. If @mfdeakin-sandia is OK with the changes to this file, then I say go for it.
Thanks, I'm just finishing up retesting now.
I'm also testing, to see if the issue I was having yesterday with the cam.h0 file has been resolved on the E3SM side. Specifically, it's marked as a file to ignore, but the test was failing despite the archiver ignoring it.

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_st_archive.py b/scripts/lib/CIME/case/case_st_archive.py
--- a/scripts/lib/CIME/case/case_st_archive.py
+++ b/scripts/lib/CIME/case/case_st_archive.py
@@ -392,10 +392,10 @@ def _archive_restarts_date_comp(case, casename, rundir, archive, archive_entry,
             restfiles = [f for f in os.listdir(rundir) if pfile.search(f)]
             print ("HERE pfile {} restfiles {}".format(pfile, restfiles))
         else:
-            pattern = r"^{}.{}[\d_]*\..*".format(casename, compname)
+            pattern = r"^{}\.{}[\d_]*\.".format(casename, compname)
             pfile = re.compile(pattern)
             files = [f for f in os.listdir(rundir) if pfile.search(f)]
-            pattern =  r'_?' + r'\d*'  +  r'\.' + suffix + r'\.' + datename_str
+            pattern =  r'_?' + r'\d*' + r'\.' + suffix + r'\.' + r'[^\.]*' + r'\.?' + datename_str
             pfile = re.compile(pattern)
             restfiles = [f for f in files if pfile.search(f)]
             logger.debug("pattern is {} restfiles {}".format(pattern, restfiles))


Test patch: