Repository: ESMCI/cime

Issue:
st_archive: DART intermediate rpointer files are rpointer.unset
Currently this issue is masked by https://github.com/ESMCI/cime/issues/1853 (which prevents DART intermediate restart / rpointer files from being archived at all); once that issue is resolved, this one will appear. As far as I can tell, this doesn't actually cause problems, but it does cause extraneous `rpointer.unset` files to appear in your archive directory.

This issue can be seen by introducing these diffs off of master: billsacks/cime@02c1616 . I plugged this into https://svn-ccsm-models.cgd.ucar.edu/clm2/trunk_tags/clm4_5_16_r253 but I think you could see the same issue from standalone cime. I then did:

```
./create_newcase --case check_clm_archiving_issue_0830b --compset I1850Clm45Bgc --res f10_f10_musgs --user-mods-dir /glade/scratch/sacks/cesm_code/clm4_5_16_r253_newCime2/components/clm/cime_config/testdefs/testmods_dirs/clm/default

cd check_clm_archiving_issue_0830b
./case.setup
./xmlchange HIST_OPTION=ndays,HIST_N=1,REST_OPTION=ndays,REST_N=1
./xmlchange DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE
./case.build && ./case.submit
```

Near the end of the short-term archiver output, I see:

```
-------------------------------------------
doing short term archiving for dart (esp)
-------------------------------------------
Archiving for date 0001-01-02-00000
writing rpointer_file /glade/scratch/sacks/archive/check_clm_archiving_issue_0830b/rest/0001-01-02-00000/rpointer.unset
Archiving for date 0001-01-03-00000
writing rpointer_file /glade/scratch/sacks/archive/check_clm_archiving_issue_0830b/rest/0001-01-03-00000/rpointer.unset
Archiving for date 0001-01-04-00000
writing rpointer_file /glade/scratch/sacks/archive/check_clm_archiving_issue_0830b/rest/0001-01-04-00000/rpointer.unset
Archiving for date 0001-01-05-00000
writing rpointer_file /glade/scratch/sacks/archive/check_clm_archiving_issue_0830b/rest/0001-01-05-00000/rpointer.unset
Archiving for date 0001-01-06-00000
```

and indeed, in my archive directory, I see a bunch of rpointer.unset files whose contents are simply `unset`.

I'm not sure how to resolve this issue. I think it arises because dart is always added to the list of compset_comps in _archive_process:

```python
    compset_comps.append('dart')
```

and its entry in config_archive.xml is (see the rpointer section near the end):

```xml
  <comp_archive_spec compclass="esp" compname="dart">
    <rest_file_extension>inflate_restart.*</rest_file_extension>
    <hist_file_extension>\.True_State.*</hist_file_extension>
    <hist_file_extension>\.Prior_Diag.*</hist_file_extension>
    <hist_file_extension>\.Posterior_Diag.*</hist_file_extension>
    <hist_file_extension>\..+\.posterior*</hist_file_extension>
    <hist_file_extension>\..+\.prior*</hist_file_extension>
    <hist_file_extension>\..+$_obs_seq.*</hist_file_extension>
    <rest_history_varname>unset</rest_history_varname>
    <rpointer>
      <rpointer_file>rpointer.unset</rpointer_file>
      <rpointer_content>unset</rpointer_content>
    </rpointer>
  </comp_archive_spec>
```

But I'm not sure what the right thing is to do here.

cc @mvertens @bertinia @jedwards4b 


Discussion:
Use unset (or something else if you don't like that) as a keyword and don't write files that contain it.
@kdraeder - @bertinia thought you might have some insight here.
I'm a bit out of my depth here (I don't work with CLM much)
but here are my thoughts.

It does seem like the 
    compset_comps.append('dart')
in _archive_process should be dependent on DATA_ASSIMILATION from env_run.xml.
Can that be done?

Farther down in the rpointer path some changes might be desirable.  
In _archive_rpointer_files any component with <rpointer> set to 'unset' 
will have rpointer files created;

```python
                if rpointer_content is not 'unset':
                    if not ninst_strings:
                        ninst_strings = ["empty"]
                for ninst_string in ninst_strings:
                    ...
```
If that 'for' loop were indented 4 spaces, to be controlled by the 'if rpointer_content' line,
then no rpointer files would be created for 'unset'.


@kdraeder Thanks a lot for these thoughts. To be clear: while I found the problem in a clm test, I actually don't think it has anything to do with clm. I'll let someone else evaluate and test your suggestion, since I'm supposed to be getting back to some other critical work.
I have not been able to reproduce the rpointer.unset problem.
My caseroot is /glade/p/work/raeder/Exp/Intermed_rpointer_unset
I built a cesm2_0_beta06 CAM, then inserted cime5.3.0-alpha.33 
into my cesm2_0_beta06 tree
/glade/p/work/raeder/Models/cesm2_0_beta06.

The relevant parts of config_archive.xml seem to be the same.

The output is (temporarily) in 
/glade2/scratch2/raeder/Intermed_rpointer_unset/{archive,run}

I'll try another run if anyone can point out what I missed
in recreating Bill's setup.

Bill; is seems like it should be a fast and helpful test to
try my indenting idea in your setup, if you still have it.

Kevin

@kdraeder it looks like your case is not using that sandbox, but is instead using an out-of-the-box beta06 sandbox, which would not exhibit this problem:

```
[cheyenne5:/glade/p/work/raeder/Exp/Intermed_rpointer_unset]$ ./xmlquery CIMEROOT
	CIMEROOT: /glade/p/cesmdata/cseg/.dev/cesm2_0_beta06/cime
[cheyenne5:/glade/p/work/raeder/Exp/Intermed_rpointer_unset]$ ./xmlquery SRCROOT
	SRCROOT: /glade/p/cesmdata/cseg/.dev/cesm2_0_beta06
```
Thanks to Bill for pointing out my blunder.
I've recreated the 'rpointer.unset' problem and found the actual cause of it.
case_st_archive.py, line 124 is
                if rpointer_content is not 'unset':
but should be 
                if rpointer_content != 'unset':
We want to compare the contents, and not whether the 2 objects are the same object.
And I still think that the indenting I suggested earlier is the best choice.
Kevin

Hi Kevin,

Thanks for tracking this down. Could you please raise an issue regarding
the problem and the solution. We can then get it into the next beta tag.

Mariana

On Mon, Sep 18, 2017 at 8:35 AM, kdraeder <notifications@github.com> wrote:

> Thanks to Bill for pointing out my blunder.
> I've recreated the 'rpointer.unset' problem and found the actual cause of
> it.
> case_st_archive.py, line 124 is
> if rpointer_content is not 'unset':
> but should be
> if rpointer_content != 'unset':
> We want to compare the contents, and not whether the 2 objects are the
> same object.
> And I still think that the indenting I suggested earlier is the best
> choice.
> Kevin
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/1854#issuecomment-330243252>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxEy_kSu6PuGwvyah-mGFQjE3tUHUoks5sjn_MgaJpZM4PIvIv>
> .
>

I've looked in several sources for how to raise an issue,
but haven't found anything.  Does it just mean start an issue?

I think that I'm solving the problem that was the source of this issue, 
so it seems redundant to raise it in an additional issue.  
However, I do have more changes in case_st_archive.py 
which need to be checked by someone else, so I could combine 
this issue (1854) and those changes into a new issue.  
What do you recommend?
Kevin
Isn't this (#1854) the appropriate issue? I think it just needs to be assigned to someone to create a PR using Kevin's solution.
@kdraeder you need to create a PR with your changes, I can walk you through how do to that.

Resolving patch:
diff --git a/scripts/lib/CIME/case_setup.py b/scripts/lib/CIME/case_setup.py
--- a/scripts/lib/CIME/case_setup.py
+++ b/scripts/lib/CIME/case_setup.py
@@ -133,7 +133,6 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
             if comp == "CPL":
                 continue
             ninst  = case.get_value("NINST_{}".format(comp))
-            # But the NINST_LAYOUT may only be concurrent in multi_driver mode
             if multi_driver:
                 expect(case.get_value("NINST_LAYOUT_{}".format(comp)) == "concurrent",
                        "If multi_driver is TRUE, NINST_LAYOUT_{} must be concurrent".format(comp))
diff --git a/scripts/lib/CIME/case_st_archive.py b/scripts/lib/CIME/case_st_archive.py
--- a/scripts/lib/CIME/case_st_archive.py
+++ b/scripts/lib/CIME/case_st_archive.py
@@ -67,8 +67,6 @@ def _get_ninst_info(case, compclass):
     for i in range(1,ninst+1):
         if ninst > 1:
             ninst_strings.append('_' + '{:04d}'.format(i))
-        else:
-            ninst_strings.append('')
 
     logger.debug("ninst and ninst_strings are: {} and {} for {}".format(ninst, ninst_strings, compclass))
     return ninst, ninst_strings
@@ -77,7 +75,7 @@ def _get_ninst_info(case, compclass):
 def _get_component_archive_entries(case, archive):
 ###############################################################################
     """
-    Each time this is generator function is called, it yields a tuple
+    Each time this generator function is called, it yields a tuple
     (archive_entry, compname, compclass) for one component in this
     case's compset components.
     """
@@ -121,28 +119,27 @@ def _archive_rpointer_files(case, archive, archive_entry, archive_restdir,
                 temp_rpointer_content = rpointer_content
                 # put in a temporary setting for ninst_strings if they are empty
                 # in order to have just one loop over ninst_strings below
-                if rpointer_content is not 'unset':
+                if rpointer_content != 'unset':
                     if not ninst_strings:
                         ninst_strings = ["empty"]
-                for ninst_string in ninst_strings:
-                    rpointer_file = temp_rpointer_file
-                    rpointer_content = temp_rpointer_content
-                    if ninst_string == 'empty':
-                        ninst_string = ""
-                    for key, value in [('$CASE', casename),
-                                       ('$DATENAME', datename),
-                                       ('$NINST_STRING', ninst_string)]:
-                        rpointer_file = rpointer_file.replace(key, value)
-                        rpointer_content = rpointer_content.replace(key, value)
-
-                    # write out the respect files with the correct contents
-                    rpointer_file = os.path.join(archive_restdir, rpointer_file)
-                    logger.info("writing rpointer_file {}".format(rpointer_file))
-                    f = open(rpointer_file, 'w')
-                    for output in rpointer_content.split(','):
-                        f.write("{} \n".format(output))
-                    f.close()
-
+                    for ninst_string in ninst_strings:
+                        rpointer_file = temp_rpointer_file
+                        rpointer_content = temp_rpointer_content
+                        if ninst_string == 'empty':
+                            ninst_string = ""
+                        for key, value in [('$CASE', casename),
+                                           ('$DATENAME', datename),
+                                           ('$NINST_STRING', ninst_string)]:
+                            rpointer_file = rpointer_file.replace(key, value)
+                            rpointer_content = rpointer_content.replace(key, value)
+       
+                        # write out the respective files with the correct contents
+                        rpointer_file = os.path.join(archive_restdir, rpointer_file)
+                        logger.debug("writing rpointer_file {}".format(rpointer_file))
+                        f = open(rpointer_file, 'w')
+                        for output in rpointer_content.split(','):
+                            f.write("{} \n".format(output))
+                        f.close()
 
 ###############################################################################
 def _archive_log_files(case, archive_incomplete, archive_file_fn):
@@ -192,15 +189,18 @@ def _archive_history_files(case, archive, archive_entry,
     rundir = case.get_value("RUNDIR")
     for suffix in archive.get_hist_file_extensions(archive_entry):
         for i in range(ninst):
-            if compname == 'dart':
-                newsuffix = casename + suffix
-            elif compname.find('mpas') == 0:
-                newsuffix = compname + '.*' + suffix
+            if ninst_string:
+               if compname.find('mpas') == 0:
+                  # Not correct, but MPAS' multi-instance name format is unknown.
+                  newsuffix = compname + '.*' + suffix
+               else:
+                  newsuffix = casename + '.' + compname + ".*" + ninst_string[i] + suffix
             else:
-                if ninst_string:
-                    newsuffix = casename + '.' + compname + ".*" + ninst_string[i] + suffix
-                else:
-                    newsuffix = casename + '.' + compname + ".*" + suffix
+               if compname.find('mpas') == 0:
+                  newsuffix = compname + '.*' + suffix
+               else:
+                  newsuffix = casename + '.' + compname + ".*" + suffix
+
             logger.debug("short term archiving suffix is {} ".format(newsuffix))
             pfile = re.compile(newsuffix)
             histfiles = [f for f in os.listdir(rundir) if pfile.search(f)]
@@ -329,21 +329,14 @@ def _archive_restarts_date_comp(case, archive, archive_entry,
                 restfiles = [f for f in os.listdir(rundir) if pfile.search(f)]
             else:
                 pattern = r"{}\.{}\d*.*".format(casename, compname)
-                if "dart" not in pattern:
-                    pfile = re.compile(pattern)
-                    files = [f for f in os.listdir(rundir) if pfile.search(f)]
-                    if ninst_strings:
-                        pattern = ninst_strings[i] + suffix + datename
-                        pfile = re.compile(pattern)
-                        restfiles = [f for f in files if pfile.search(f)]
-                    else:
-                        pattern = suffix + datename
-                        pfile = re.compile(pattern)
-                        restfiles = [f for f in files if pfile.search(f)]
+                pfile = re.compile(pattern)
+                files = [f for f in os.listdir(rundir) if pfile.search(f)]
+                if ninst_strings:
+                    pattern = ninst_strings[i] + suffix + datename
                 else:
-                    pattern = suffix
-                    pfile = re.compile(pattern)
-                    restfiles = [f for f in os.listdir(rundir) if pfile.search(f)]
+                    pattern = suffix + datename
+                pfile = re.compile(pattern)
+                restfiles = [f for f in files if pfile.search(f)]
 
             for restfile in restfiles:
                 restfile = os.path.basename(restfile)
@@ -364,21 +357,20 @@ def _archive_restarts_date_comp(case, archive, archive_entry,
                     srcfile = os.path.join(rundir, restfile)
                     destfile = os.path.join(archive_restdir, restfile)
                     last_restart_file_fn(srcfile, destfile)
-                    logger.info("{} \n{} to \n{}".format(
-                        last_restart_file_fn_msg, srcfile, destfile))
+                    logger.info("{} {} \n{} to \n{}".format(
+                        "datename_is_last", last_restart_file_fn_msg, srcfile, destfile))
                     for histfile in histfiles_for_restart:
                         srcfile = os.path.join(rundir, histfile)
                         destfile = os.path.join(archive_restdir, histfile)
                         expect(os.path.isfile(srcfile),
                                "restart file {} does not exist ".format(srcfile))
                         shutil.copy(srcfile, destfile)
-                        logger.info("copying \n{} to \n{}".format(srcfile, destfile))
+                        logger.info("datename_is_last + histfiles_for_restart copying \n{} to \n{}".format(srcfile, destfile))
                 else:
                     # Only archive intermediate restarts if requested - otherwise remove them
                     if case.get_value('DOUT_S_SAVE_INTERIM_RESTART_FILES'):
                         srcfile = os.path.join(rundir, restfile)
                         destfile = os.path.join(archive_restdir, restfile)
-                        logger.info("moving \n{} to \n{}".format(srcfile, destfile))
                         expect(os.path.isfile(srcfile),
                                "restart file {} does not exist ".format(srcfile))
                         archive_file_fn(srcfile, destfile)


Test patch: