Repository: ESMCI/cime

Issue:
case_st_archive.py and env_archive.xml regexes in the context of DART
@jedwards4b @gold2718 @mvertens 

Some of the discussion in #2308 about changing regular expressions
in st_archive may be relevant to this issue.

While developing the ability to st_archive the files
created by DART, it was difficult to understand how
to use env_archive.xml.  (I have since stumbled across
www.cesm.ucar.edu/models/cesm2.0/cesm/env_archive.html,
but others might not find it, since there's no link to it
that I can find.)
There are several related issues.
1) The use of regular expression metacharacters
   makes it harder to understand.  There are no
   comments to note that they are python regexes.
   There's not enough context in the env_archive.xml
   file to understand what they need to do.
2) The inconsistency between the need for '.' separators
   in the <rest_file_extension> 'suffixes' but not
   in the <hist_file_extension> 'suffixes' because
   _archive_history_files inserts them
   (or tries to; see item 3).
3) Quite a few regular expressions that *are* in
   case_st_archive.py seem to not do quite what was
   intended.  They worked in many cases, but not as
   robustly or selectively as they should.
   These mostly involved combinations of ., \., \d and *.

I've modified scripts and xml files to move many
of the metacharacters out of env_archive.xml and
into case_st_archive.xml.
To see how this cleans up env_batch.xml, first look at
   config/config_headers.xml     (additional header comments for users)
   config/cesm/config_archive.xml
   src/drivers/mct/cime_config/config_archive.xml
   src/components/data_comps/docn/cime_config/config_archive.xml
Then look at case_st_archive.py.

I've attached a tar file of a representative CAM+POP+DART $rundir
if someone wants it for testing st_archive.  The files in it are just
placeholders, so it can be copied fast.
[CAM+POP+DART_run.gztar.zip](https://github.com/ESMCI/cime/files/1772851/CAM.POP.DART_run.gztar.zip)




Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case_st_archive.py b/scripts/lib/CIME/case_st_archive.py
--- a/scripts/lib/CIME/case_st_archive.py
+++ b/scripts/lib/CIME/case_st_archive.py
@@ -199,6 +199,7 @@ def _get_component_archive_entries(case, archive):
     compset_comps.append('dart')
 
     for compname in compset_comps:
+        logger.debug("compname is {} ".format(compname))
         archive_entry = archive.get_entry(compname)
         if archive_entry is not None:
             yield(archive_entry, compname, archive.get(archive_entry, "compclass"))
@@ -313,19 +314,20 @@ def _archive_history_files(case, archive, archive_entry,
             if ninst_string:
                 if compname.find('mpas') == 0:
                     # Not correct, but MPAS' multi-instance name format is unknown.
-                    newsuffix = compname + '.*' + suffix
+                    newsuffix =                    compname + r'\d*'                   + r'\.' + suffix + r'\.'
                 else:
-                    newsuffix = casename + '.' + compname + ".*" + ninst_string[i] + suffix
+                    newsuffix = casename + r'\.' + compname + r'\d*' + ninst_string[i] + r'\.' + suffix + r'\.'
             else:
                 if compname.find('mpas') == 0:
-                    newsuffix = compname + '.*' + suffix
+                    newsuffix =                    compname + r'\d*'                   + r'\.' + suffix + r'\.'
                 else:
-                    newsuffix = casename + '.' + compname + ".*" + suffix
-
+                    newsuffix = casename + r'\.' + compname + r'\d*'                   + r'\.' + suffix + r'\.'
             logger.debug("short term archiving suffix is {} ".format(newsuffix))
 
             pfile = re.compile(newsuffix)
             histfiles = [f for f in os.listdir(rundir) if pfile.search(f)]
+            logger.debug("histfiles = {} ".format(histfiles))
+
             if histfiles:
                 for histfile in histfiles:
                     file_date = get_file_date(os.path.basename(histfile))
@@ -465,16 +467,16 @@ def _archive_restarts_date_comp(case, archive, archive_entry,
                 pfile = re.compile(pattern)
                 restfiles = [f for f in os.listdir(rundir) if pfile.search(f)]
             else:
-                pattern = r"{}\.{}\d*.*".format(casename, compname)
+                pattern = r"{}.{}[\d_]*\..*".format(casename, compname)
                 pfile = re.compile(pattern)
                 files = [f for f in os.listdir(rundir) if pfile.search(f)]
                 if ninst_strings:
-                    pattern = ninst_strings[i] + suffix + datename_str
+                    pattern =  ninst_strings[i] + r'\.' + suffix + r'\.' + datename_str
                 else:
-                    pattern = suffix + datename_str
+                    pattern =                     r'\.' + suffix + r'\.' + datename_str
                 pfile = re.compile(pattern)
                 restfiles = [f for f in files if pfile.search(f)]
-            logger.debug("Pattern is {} restfiles {}".format(pattern, restfiles))
+                logger.debug("pattern is {} restfiles {}".format(pattern, restfiles))
             for restfile in restfiles:
                 restfile = os.path.basename(restfile)
 


Test patch: