Repository: ESMCI/cime

Issue:
Should tests' hist_utils move rather than copying history files?
I noticed that `hist_utils.py : move` actually copies rather than moving the last set of history files (despite the name). For tests where two runs are done out of the same run directory (such as ERS), this can cause it to look like the second run produced a history file when it really did not. 

For example, imagine that the first run produced a history file named `foo.h.0001-01.nc` - and this was the only history file produced for the 'foo' component. Then `move` would copy this to `foo.h.0001-01.nc.base`. Now imagine that the second run failed to produce any history files. hist_utils would look for foo's history files in the run directory, find the still-existing `foo.h.0001-01.nc` (which was produced by the first run), and copy it to `foo.h.0001-01.nc.rest` (or whatever suffix is used for the second run). Then, of course the history file comparison would match, because this is really the same file as the .base file.

I think the only thing needed to make the tests more robust would be to change `shutil.copy` to `os.rename` in the `move` function. However, to avoid confusion, an ideal behavior to me would be for `move` to move _all_ history files (i.e., everything found by `_get_all_hist_files`) to some subdirectory (e.g., `"saved_history_files_%s"%suffix`), and/or append the suffix to the end of all history files. Then if a second run is done out of the same case directory, there will be no possible confusion as to which history files were produced by each run. Without the second piece, I think we could get confusing results, but probably wouldn't get any accidental passes. For example, if the first run produced `foo.h.0001-01.nc` and `foo.h.0001-02.nc` and the second run produced no history files, then, with `shutil.copy` changed to `os.rename`, we'd get `foo.h.0001-02.nc.base` and `foo.h.0001-01.nc.rest`. There shouldn't be any `.rest` file from this run, so that's confusing - but at least it would still be reported as a failure.

However, looking back at the cime4 version of `component_compare_move.sh`, I see that there was the same behavior / misnomer, along with this comment from @bandre-ucar :

```
        # FIXME(bja, 2014-11) temp change to test if cp fixes the
        # problem with mfilt in restart tests. If it does, then
        # rename this script to component_compare_copy.sh !
        /bin/cp "$test_hist" "$test_hist_suffix"
```

@bandre-ucar or @mvertens - can you remember the details of the problem that prompted the change to cp rather than mv?

If there is a legitimate reason why this needs to be a copy rather than a move (as suggested by the above comment from bja), then this could cause problems for the clone-based test infrastructure (SystemTestCompareTwo), if we switch restart tests like ERP to use this infrastructure: For these tests, there is a clear separation between the files produced by each run, so the behavior is similar to what I describe as an "ideal behavior" above, and would presumably trigger whatever problem prompted bja to make the switch to /bin/cp.

Tagging @mvertens @jedwards4b @jgfouca @bandre-ucar 



Discussion:
To clarify what I mean about this:

> However, to avoid confusion, an ideal behavior to me would be for move to move all history files (i.e., everything found by _get_all_hist_files) to some subdirectory (e.g., "saved_history_files_%s"%suffix), and/or append the suffix to the end of all history files. 

I still imagine that _comparisons_ would just be done using the last history file from each component (as is the current behavior), so it's possible that some other code would have to be tweaked a bit to handle the fact that we'd now have a whole bunch of `*.base` files (or whatever), but still just want to do comparisons using the last from each component. (Or maybe the logic already handles that... I haven't looked.)

The reason you need to copy rather than move is if you have a history file
that will need to continue to be filled on the next phase (say a restart
run). If this is not the case, then a move is fine. Maybe the best solution
would be to have the option to either copy or move based as an input
argument that each test can set separately.

On Sat, Sep 24, 2016 at 6:07 AM, Bill Sacks notifications@github.com
wrote:

> I noticed that hist_utils.py : move actually copies rather than moving
> the last set of history files (despite the name). For tests where two runs
> are done out of the same run directory (such as ERS), this can cause it to
> look like the second run produced a history file when it really did not.
> 
> For example, imagine that the first run produced a history file named
> foo.h.0001-01.nc - and this was the only history file produced for the
> 'foo' component. Then move would copy this to foo.h.0001-01.nc.base. Now
> imagine that the second run failed to produce any history files. hist_utils
> would look for foo's history files in the run directory, find the
> still-existing foo.h.0001-01.nc (which was produced by the first run),
> and copy it to foo.h.0001-01.nc.rest (or whatever suffix is used for the
> second run). Then, of course the history file comparison would match,
> because this is really the same file as the .base file.
> 
> I think the only thing needed to make the tests more robust would be to
> change shutil.copy to os.rename in the move function. However, to avoid
> confusion, an ideal behavior to me would be for move to move _all_
> history files (i.e., everything found by _get_all_hist_files) to some
> subdirectory (e.g., "saved_history_files_%s"%suffix), and/or append the
> suffix to the end of all history files. Then if a second run is done out of
> the same case directory, there will be no possible confusion as to which
> history files were produced by each run. Without the second piece, I think
> we could get confusing results, but probably wouldn't get any accidental
> passes. For example, if the first run produced foo.h.0001-01.nc and
> foo.h.0001-02.nc and the second run produced no history files, then, with
> shutil.copy changed to os.rename, we'd get foo.h.0001-02.nc.base and
> foo.h.0001-01.nc.rest. There shouldn't be any .rest file from this run,
> so that's confusing - but at least it would still be reported as a failure.
> 
> However, looking back at the cime4 version of component_compare_move.sh,
> I see that there was the same behavior / misnomer, along with this comment
> from @bandre-ucar https://github.com/bandre-ucar :
> 
> ```
>     # FIXME(bja, 2014-11) temp change to test if cp fixes the
>     # problem with mfilt in restart tests. If it does, then
>     # rename this script to component_compare_copy.sh !
>     /bin/cp "$test_hist" "$test_hist_suffix"
> ```
> 
> @bandre-ucar https://github.com/bandre-ucar or @mvertens
> https://github.com/mvertens - can you remember the details of the
> problem that prompted the change to cp rather than mv?
> 
> If there is a legitimate reason why this needs to be a copy rather than a
> move (as suggested by the above comment from bja), then this could cause
> problems for the clone-based test infrastructure (SystemTestCompareTwo), if
> we switch restart tests like ERP to use this infrastructure: For these
> tests, there is a clear separation between the files produced by each run,
> so the behavior is similar to what I describe as an "ideal behavior" above,
> and would presumably trigger whatever problem prompted bja to make the
> switch to /bin/cp.
> 
> Tagging @mvertens https://github.com/mvertens @jedwards4b
> https://github.com/jedwards4b @jgfouca https://github.com/jgfouca
> @bandre-ucar https://github.com/bandre-ucar
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/ESMCI/cime/issues/583, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AHlxE26Rl5Qzpm_c10AlP0lRO43TYBMJks5qtRKagaJpZM4KFoLf
> .

@mvertens that's a very good point. Now I understand.

> Maybe the best solution
> would be to have the option to either copy or move based as an input
> argument that each test can set separately.

Since we want to move most of the non-restart tests to the SystemTestCompareTwo infrastructure, this is probably a moot point.

So maybe the solution here is to just rename the `move` function to `copy`, and to add a comment describing the need for copying, similar to Mariana's post? That would help prevent this question from coming up again in the future. (Otherwise, I'm sure I'll ask it again a year from now once I've forgotten this discussion.)

Do others feel like that's a reasonable solution?

@billsacks yes, i like that solution

Okay, I'll make this change soon.


Resolving patch:
diff --git a/utils/python/CIME/SystemTests/eri.py b/utils/python/CIME/SystemTests/eri.py
--- a/utils/python/CIME/SystemTests/eri.py
+++ b/utils/python/CIME/SystemTests/eri.py
@@ -191,7 +191,7 @@ def run_phase(self):
             newfile = os.path.basename(newfile)
             os.symlink(item, os.path.join(rundir, newfile))
 
-        self._component_compare_move("hybrid")
+        self._component_compare_copy("hybrid")
 
         # run branch case (short term archiving is off)
         self.run_indv()
diff --git a/utils/python/CIME/hist_utils.py b/utils/python/CIME/hist_utils.py
--- a/utils/python/CIME/hist_utils.py
+++ b/utils/python/CIME/hist_utils.py
@@ -49,11 +49,11 @@ def _get_latest_hist_files(testcase, model, from_dir, suffix=""):
     return histlist
 
 
-def move(case, suffix):
-    """
-    Change the suffix for the most recent batch of hist files in a case. This can
-    allow you to temporarily "save" these files so they won't be blown away if you
-    re-run the case.
+def copy(case, suffix):
+    """Copy the most recent batch of hist files in a case, adding the given suffix.
+
+    This can allow you to temporarily "save" these files so they won't be blown
+    away if you re-run the case.
 
     case - The case containing the files you want to save
     suffix - The string suffix you want to add to saved files, this can be used to find them later.
@@ -62,21 +62,34 @@ def move(case, suffix):
     testcase = case.get_value("CASE")
 
     # Loop over models
-    comments = "Moving hist files to suffix '%s'\n" % suffix
-    num_moved = 0
+    comments = "Copying hist files to suffix '%s'\n" % suffix
+    num_copied = 0
     for model in _iter_model_file_substrs(case):
-        comments += "  Moving hist files for model '%s'\n" % model
+        comments += "  Copying hist files for model '%s'\n" % model
         test_hists = _get_latest_hist_files(testcase, model, rundir)
-        num_moved += len(test_hists)
+        num_copied += len(test_hists)
         for test_hist in test_hists:
             new_file = "%s.%s" % (test_hist, suffix)
             if os.path.exists(new_file):
                 os.remove(new_file)
 
             comments += "    Copying '%s' to '%s'\n" % (test_hist, new_file)
+
+            # Need to copy rather than move in case there are some history files
+            # that will need to continue to be filled on the next phase; this
+            # can be the case for a restart run.
+            #
+            # (If it weren't for that possibility, a move/rename would be more
+            # robust here: The problem with a copy is that there can be
+            # confusion after the second run as to which files were created by
+            # the first run and which by the second. For example, if the second
+            # run fails to output any history files, the test will still pass,
+            # because the test system will think that run1's files were output
+            # by run2. But we live with that downside for the sake of the reason
+            # noted above.)
             shutil.copy(test_hist, new_file)
 
-    expect(num_moved > 0, "move failed: no hist files found in rundir '%s'" % rundir)
+    expect(num_copied > 0, "copy failed: no hist files found in rundir '%s'" % rundir)
 
     return comments
 


Test patch:
diff --git a/scripts/Testing/Testcases/NOC_script b/scripts/Testing/Testcases/NOC_script
--- a/scripts/Testing/Testcases/NOC_script
+++ b/scripts/Testing/Testcases/NOC_script
@@ -75,10 +75,10 @@ else
 endif
 
 echo "moving relevant history files to suffix with command " >>& $TESTSTATUS_LOG
-echo "$SCRIPTSROOT/Tools/component_compare_move base" >>& $TESTSTATUS_LOG
+echo "$SCRIPTSROOT/Tools/component_compare_copy base" >>& $TESTSTATUS_LOG
 echo "" >>& $TESTSTATUS_LOG
 
-$SCRIPTSROOT/Tools/component_compare_move base
+$SCRIPTSROOT/Tools/component_compare_copy base
 
 #======================================================================
 # do an initial run test with NINST 2, with the
@@ -117,10 +117,10 @@ else
 endif
 
 echo "moving relevant history files to suffix with commands " >>& $TESTSTATUS_LOG
-echo "$SCRIPTSROOT/Tools/component_compare_move inst2mod"  >>& $TESTSTATUS_LOG
+echo "$SCRIPTSROOT/Tools/component_compare_copy inst2mod"  >>& $TESTSTATUS_LOG
 echo "" >>& $TESTSTATUS_LOG
 
-$SCRIPTSROOT/Tools/component_compare_move inst2mod
+$SCRIPTSROOT/Tools/component_compare_copy inst2mod
 
 #======================================================================
 # Check test status for all relevant component history files
diff --git a/utils/python/CIME/SystemTests/system_tests_common.py b/utils/python/CIME/SystemTests/system_tests_common.py
--- a/utils/python/CIME/SystemTests/system_tests_common.py
+++ b/utils/python/CIME/SystemTests/system_tests_common.py
@@ -206,7 +206,7 @@ def run_indv(self, suffix="base", st_archive=False):
             expect(False, "Coupler did not indicate run passed")
 
         if suffix is not None:
-            self._component_compare_move(suffix)
+            self._component_compare_copy(suffix)
 
         if st_archive:
             case_st_archive(self._case)
@@ -222,8 +222,8 @@ def _coupler_log_indicates_run_complete(self):
             logger.info("%s is not compressed, assuming run failed"%newestcpllogfile)
         return False
 
-    def _component_compare_move(self, suffix):
-        comments = move(self._case, suffix)
+    def _component_compare_copy(self, suffix):
+        comments = copy(self._case, suffix)
         append_status(comments, sfile="TestStatus.log")
 
     def _component_compare_test(self, suffix1, suffix2):