Repository: ESMCI/cime

Issue:
Remove any existing cprnc output files before running cprnc
I should preface this by saying that this showed up for me in a rather screwey test case. I'm not 100% sure this will show up in all testing, but looking at what I think is the offending code, I think it would.

I ran a system test that did baseline comparisons; this passed (including the baseline comparisons - it was bfb with baselines). Then I made a change that caused the results to differ from the baselines and reran the test. To my surprise, this reported that the BASELINE phase PASSed.

Digging into this, it looks like the output from cprnc is appended to the end of any existing cprnc.out file, rather than overwriting the old file. Then this whole file (now containing multiple sets of output) is read back in, and the first thing the code looks for is an occurrence of "files seem to be IDENTICAL". Here is the relevant code:

https://github.com/ESMCI/cime/blob/13a8d5c5e89cc10719a0b23acf86331ab83c9448/scripts/lib/CIME/hist_utils.py#L346-L374

In this case, both "files seem to be IDENTICAL" and "the two files seem to be DIFFERENT" appear in the output file; the code treats this as being identical, even though the most recent run led to a difference.

I'm labeling this as CRITICAL because I view any bug that causes tests to be reported as a PASS when they should be a FAIL to be a critical issue.

I don't have time to fix this right now, given other things that I'm scrambling to finish before taking a few days off. But I'd suggest:

- Definitely: removing any existing file (i.e., the `output_filename`) before running cprnc, so that only the most recent output appears in there. (If the appending was by design, then the old file could be saved away somewhere rather than just removed.)

- Possibly: reordering the logic in the bottom half of the above code snippet: The safest thing to do is to first check for any failure conditions and then finally check for "files seem to be IDENTICAL" as the last check. This way, if we do run into a situation where both "DIFFERENT" and "IDENTICAL" appear in the output, we err on the side of treating this as DIFFERENT, which seems safest.

@jgfouca @jedwards4b @WesCoomber 


Discussion:
Nice find! I think I can fix it fairly easily.
@billsacks , the changes you want are easy, but I thought it would be a good idea to make a new test to catch this error. I was surprised to see that everything worked fine for me without any changes:

```
% ./create_test -g -b fouc TESTRUNDIFF_P1.f19_g16_rx1.A --wait
% ./create_test -c -b fouc TESTRUNDIFF_P1.f19_g16_rx1.A --wait
% cd $2nd-casedir
% TESTRUNDIFF_ALTERNATE=True ./case.submit
% cat TestStatus | grep BASELINE
FAIL TESTRUNDIFF_P1.f19_g16_rx1.A.mappy_gnu BASELINE fouc: DIFF
```

Two potential factors:
1) I have a brand-new build of cprnc
2) I'm on a desktop, not batch machine

I'd be curious if you can get a simple sequence of commands to demonstrate the bug.
@jgfouca okay, I see what's going on here, and I really apologize for making this seem like a bigger issue than it turns out to be! It looks like the cprnc.out files are generally removed in SystemTestCommon's `run_indv`, but my somewhat screwey test case doesn't use `run_indv`, because it doesn't use `case.run`. So my new test case is quite likely the only test case for which this is a problem.

I guess my feeling is: if you have already done essentially all the work that would need to be done to put this file removal in the cprnc python function, then that still feels most robust, for the sake of anything that calls cprnc outside of SystemTestCommon's `run_indv`. But if this would take more work on your part, then I'm fine with dropping this issue and I'll just add the necessary removal to my new system test.

In any case, I'll rename this issue to be more accurate. Again, sorry about that.
@jgfouca I'll also add another option: Since I'm under less time pressure now: if there is some additional work to do on this, and/or some manual testing to verify correctness, I'd be happy to help finish up the work on fixing this issue by changing the python cprnc code.
@billsacks , that's OK. I already did that part.

Resolving patch:
diff --git a/scripts/lib/CIME/hist_utils.py b/scripts/lib/CIME/hist_utils.py
--- a/scripts/lib/CIME/hist_utils.py
+++ b/scripts/lib/CIME/hist_utils.py
@@ -346,6 +346,10 @@ def cprnc(model, file1, file2, case, rundir, multiinst_driver_compare=False, out
     if outfile_suffix is None:
         cpr_stat, out, _ = run_cmd("{} -m {} {}".format(cprnc_exe, file1, file2), combine_output=True)
     else:
+        # Remove existing output file if it exists
+        if os.path.exists(output_filename):
+            os.remove(output_filename)
+
         cpr_stat = run_cmd("{} -m {} {}".format(cprnc_exe, file1, file2), combine_output=True, arg_stdout=output_filename)[0]
         with open(output_filename, "r") as fd:
             out = fd.read()
@@ -360,9 +364,7 @@ def cprnc(model, file1, file2, case, rundir, multiinst_driver_compare=False, out
             # have no differences.
             files_match = " 0 had non-zero differences" in out
         else:
-            if "files seem to be IDENTICAL" in out:
-                files_match = True
-            elif "the two files seem to be DIFFERENT" in out:
+            if "the two files seem to be DIFFERENT" in out:
                 files_match = False
             elif "the two files DIFFER only in their field lists" in out:
                 if ignore_fieldlist_diffs:
@@ -370,11 +372,14 @@ def cprnc(model, file1, file2, case, rundir, multiinst_driver_compare=False, out
                 else:
                     files_match = False
                     comment = CPRNC_FIELDLISTS_DIFFER
+            elif "files seem to be IDENTICAL" in out:
+                files_match = True
             else:
                 expect(False, "Did not find an expected summary string in cprnc output:\n{}".format(out))
     else:
         # If there is an error in cprnc, we do the safe thing of saying the comparison failed
         files_match = False
+
     return (files_match, output_filename, comment)
 
 def compare_baseline(case, baseline_dir=None, outfile_suffix=""):


Test patch: