Repository: ESMCI/cime

Issue:
Change provenance checking from stop on error, to simply write a warning...
The call to manage_externals to establish status for provenance information requires success. I think this should allow it to fail, and for it to report it had trouble finding the info.

@rgknox did have trouble with this. We have asked him to replicate the problem.

Here's the current code in CIME/provenance.py
```
def _save_build_provenance_cesm(case, lid): # pylint: disable=unused-argument
    version = case.get_value("MODEL_VERSION")
    # version has already been recorded
    srcroot = case.get_value("SRCROOT")
    manic = os.path.join(srcroot, "manage_externals","checkout_externals")
    out = None
    if os.path.exists(manic):
        out = run_cmd_no_fail(manic + " --status --verbose", from_dir=srcroot)
```

replacing run_cmd_no_fail with "run_cmd" would have it NOT die on an error, also give some information about the error it ran into.

Leaving it as it is, means we require it to work no matter what. Changing it does mean that sometimes the provenance information wouldn't be correct if something went wrong. So there is more than one way to look at this. So I'd like some discussion and then we can set it the way that we think makes the most sense. 



Discussion:
Can you open an issue in manage_externals and document what caused it to fail.  I've been trying to make sure that this doesn't happen.   Also make sure that this is actually broken at the head of the manage_externals master - some problems have already been addressed. 
Here's the error that @rgknox got below. From looking at it, it's because his Externals file was screwed up. I think he had a branch name for the tag keyword. So again the question is do we want it to fail when there is some problem like this in finding provenance information because a manage_externals file is screwed up. Or do we want it to go on and just report a problem?
```
Finished MODEL_BUILD for test ERS_D_Ld5.f19_g16.I2000Clm50BgcCruGs.lawrencium-lr3_intel.clm-default in 207.398784 seconds (FAIL). [COMPLETED 1 of 16]
    Case dir: /global/scratch/rgknox/ctsm-tests/fates.lawrencium-lr3.intel.Cd484ca2-F5c40b26-ngeet-master/ERS_D_Ld5.f19_g16.I2000Clm50BgcCruGs.lawrencium-lr3_intel.clm-default.G.20180315_150625_084n23
    Errors were:
        Building test for ERS in directory /global/scratch/rgknox/ctsm-tests/fates.lawrencium-lr3.intel.Cd484ca2-F5c40b26-ngeet-master/ERS_D_Ld5.f19_g16.I2000Clm50BgcCruGs.lawrencium-lr3_intel.clm-default.G.20180315_150625_084n23
        Exception during build:
        ERROR: Command: '/global/scratch/rgknox/ctsm/manage_externals/checkout_externals --status --verbose' failed with error 'Processing externals description file : Externals.cfg
        Processing externals description file : Externals_CISM.cfg
        Processing externals description file : Externals_CLM.cfg
        Checking status of externals: cism, source_cism, clm, fates, 
        ERROR: DEV_ERROR: regex to detect tracking branch failed.' from dir '/global/scratch/rgknox/ctsm'
        Traceback (most recent call last):
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 88, in build
            model_only=(phase_name==MODEL_BUILD_PHASE))
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 119, in build_phase
            self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 126, in build_indv
            sharedlib_only=sharedlib_only, model_only=model_only)
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 584, in case_build
            return run_and_log_case_status(functor, "case.build", caseroot=caseroot)
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1496, in run_and_log_case_status
            rv = func()
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 583, in <lambda>
            functor = lambda: _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist)
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 533, in _case_build_impl
            post_build(case, logs, build_complete=not (buildlist or sharedlib_only))
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 578, in post_build
            save_build_provenance(case, lid=lid)
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/provenance.py", line 102, in save_build_provenance
            _save_build_provenance_cesm(case, lid)
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/provenance.py", line 86, in _save_build_provenance_cesm
            out = run_cmd_no_fail(manic + " --status --verbose", from_dir=srcroot)
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 424, in run_cmd_no_fail
            expect(False, "Command: '{}' failed with error '{}' from dir '{}'".format(cmd, errput.encode('utf-8'), os.getcwd() if from_dir is None else from_dir))
          File "/global/scratch/rgknox/ctsm/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 112, in expect
            raise exc_type(msg)
        SystemExit: ERROR: Command: '/global/scratch/rgknox/ctsm/manage_externals/checkout_externals --status --verbose' failed with error 'Processing externals description file : Externals.cfg
        Processing externals description file : Externals_CISM.cfg
        Processing externals description file : Externals_CLM.cfg
        Checking status of externals: cism, source_cism, clm, fates, 
        ERROR: DEV_ERROR: regex to detect tracking branch failed.' from dir '/global/scratch/rgknox/ctsm'
```
It feels like there are two questions:

(1) (Should be moved to manage_externals issue): Under what circumstances, if any, should `checkout_externals --status` give an error?

(2) What should the provenance code do when `checkout_externals` gives an error?

I feel like it's going to be hard / undesirable to completely eliminate the possibility of errors from `checkout_externals --status`. For example, what if there is a totally garbage `Externals.cfg` file? So (2) will be relevant in some cases. I personally don't have strong feelings as to whether there should be a warning or an error in these cases – it all depends on how important we feel it is that provenance information be captured: if it's important, then this should result in an error.
(@ekluzek and @rgknox I know my current feelings don't totally match what I said on Thursday, when I think I supported turning this into a warning. Upon further reflection I'm thinking that maybe an error really is appropriate here - but again, I don't have strong feelings about this.)
I can go either way on this. I do think for important simulations I would like the provenance to be there for the case. So there are times I would want it to error out. But, it's also not obvious that you can't run a case or a test because the externals file for manage_externals are screwed up in some manner. And for a test or most debugging simulations I don't care if the provenance information isn't set.

I created an issue for this in manage_externals..

https://github.com/NCAR/manage_externals/issues/89
@ekluzek I feel it's important for `manage_externals` to return an error code in some cases, like when the externals file is really messed up - i.e., saying, "I was not able to successfully determine the status". Then it's up to the caller to determine how to handle this. The provenance function could decide to proceed despite the error, or it could be changed to give a more meaningful error message if you feel that would help resolve this problem.
Wait - this is a legitimate error then - I think that manage_externals and create_newcase did the right thing here.
Hi All, 

This was the original problem I "thought" I had, and the steps I "thought" I took:
1) cloned ctsm
2) executed ./manage_externals/checkout_externals
3) changed in src/fates, then checked out a new branch from a remote
4) ran a test

My thoughts at the time, were that the testing system was calling manage_externals to compare what it had in the configuration against what was actually checked out, and that there was some conflict with the new branch I checked-out inside fates.

However, whatever problem I was having, was probably just me making a mistake.  I just tried those same steps and the test ran just fine.

My understanding now, is that if I use git to directly check-out new branches inside src/fates, then I do not have to make updates to the fates entry in Externals_CLM.cfg to run tests.

Sorry for the confusion from my side.






We talked about this briefly in a CSEG meeting, and decided to leave the current behavior in place. It is legit for manage_externals to die on a legitimate error. This does mean that users have to keep manage_externals functioning to at least give a valid status. So I think this means as part of the workflow to run "manage_externals" "--status" whenever you make changes that might screw it up so that your cases don't die when you create them.
Because of the placement of the call to gather provenance information (at the very end of `post_build` in `build.py`), a failure in gathering provenance information results in an error being recorded on stdout and in CaseStatus, but BUILD_COMPLETE is still set to TRUE, so you're allowed to continue on to submit the run. I'd argue that this behavior is undesirable:

1. If our intention is to ensure that provenance information is always gathered successfully before allowing the run to proceed, the current implementation does not accomplish this, because BUILD_COMPLETE is set to TRUE.

2. If our intention is to allow a run to proceed even if we fail to gather provenance information, the current implementation is confusing for users who see an error message printed at the end of the build rather than seeing, "MODEL BUILD HAS FINISHED SUCCESSFULLY".

I'm starting to feel that (like @ekluzek originally suggested), a manage_externals failure should only result in a warning, not an error. This feeling is based on seeing another possible reason why this can fail: if you originally ran manage_externals with an older version of svn, but now have a newer version installed. And I wouldn't be surprised if there are some other scenarios that can cause a failure in manage_externals. I feel like it's going to be frustrating to users to experience these errors when building a case.

That said: I'd still be okay with giving a full-on error if the provenance function fails if people feel that's the best behavior. But in that case, I feel this should mark BUILD_COMPLETE as FALSE, so that it actually stops you from running the model.
@jedwards4b and @mvertens I'm labeling this with the cesm2 milestone: I feel like we should get something more consistent in place in this respect (see my previous comment).
Presuming `case.build` and `case.test` are the only user-facing script that calls `_save_build_provenance_cesm`, I'm in favor of the following:

1. `manage_externals` continues to return errors when it deems it appropriate
2. CESM does not set `BUILD_COMPLETE=TRUE` if `_save_build_provenance_cesm` fails
3. The build and test scripts have a `--skip-provenance-check` flag that skips this check (and maybe also let users set `SKIP_PROVENANCE_CHECK` in `.cime/config`?)

There are instances where we absolutely need to be sure that the provenance being reported is correct (otherwise why have the feature?), and for those cases we need `case.build` to prevent the user from continuing. There are other times when developers know their sandbox doesn't match what's in `Externals.cfg`, and they're just testing a new feature, so why would you even want the provence check to be run?
For clarity the provenance check is not looking at Externals.cfg, it's querying repos to find out exactly what you do have.    Otherwise I agree with the approach @mnlevy1981 proposes.
comments from a user (me)...

Using a relatively recent codebase (cesm2_0_exp10e), I've cloned a case of
Cecile that uses a CESM sandbox of hers. When I build, I get an error from
checkout_externals. It appears that checkout_externals is attempting to
write to a mange_externals.log file that resides within her sandbox, and I
don't have write permission on that file.

I'm glad that the error is not truly fatal. Otherwise I would be unable to
easily clone her case.

So please consider the usecase of building a case when you don't have write
permission to $CIMEROOT.

FYI, the error message is:

ERROR: Command:
'/glade/p/work/hannay/cesm_tags/cesm2_0_exp10e/cime/../manage_externals/checkout_externals
--status --verbose' failed with error '[Errno 13] Permission denied:
u'/glade/p/work/hannay/cesm_tags/cesm2_0_exp10e/manage_externals.log'' from
dir '/glade/p/work/hannay/cesm_tags/cesm2_0_exp10e/cime/..'


On Thu, May 3, 2018 at 3:43 PM, jedwards4b <notifications@github.com> wrote:

> For clarity the provenance check is not looking at Externals.cfg, it's
> querying repos to find out exactly what you do have. Otherwise I agree with
> the approach @mnlevy1981 <https://github.com/mnlevy1981> proposes.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/2430#issuecomment-386446662>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AO2Xu8ggLLqXaCKk6iaUocirBqxhbHVCks5tu3oDgaJpZM4TKzNt>
> .
>

I like @mnlevy1981 's suggestions. The error printed if the provenance-gathering fails should suggest re-running with the `--skip-provenance-check` flag if you don't care about provenance information.

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -4,7 +4,7 @@
 import glob, shutil, time, threading, subprocess
 from CIME.XML.standard_module_setup  import *
 from CIME.utils                 import get_model, analyze_build_log, stringify_bool, run_and_log_case_status, get_timestamp, run_sub_or_cmd, run_cmd, get_batch_script_for_job, gzip_existing_file, safe_copy
-from CIME.provenance            import save_build_provenance
+from CIME.provenance            import save_build_provenance as save_build_provenance_sub
 from CIME.locked_files          import lock_file, unlock_file
 
 logger = logging.getLogger(__name__)
@@ -364,7 +364,8 @@ def _clean_impl(case, cleanlist, clean_all, clean_depends):
     case.flush()
 
 ###############################################################################
-def _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist):
+def _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist,
+                     save_build_provenance):
 ###############################################################################
 
     t1 = time.time()
@@ -528,7 +529,8 @@ def _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist):
             case.read_xml()
             # Note, doing buildlists will never result in the system thinking the build is complete
 
-    post_build(case, logs, build_complete=not (buildlist or sharedlib_only))
+    post_build(case, logs, build_complete=not (buildlist or sharedlib_only),
+               save_build_provenance=save_build_provenance)
 
     t3 = time.time()
 
@@ -540,13 +542,16 @@ def _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist):
     return True
 
 ###############################################################################
-def post_build(case, logs, build_complete=False):
+def post_build(case, logs, build_complete=False, save_build_provenance=True):
 ###############################################################################
     for log in logs:
         gzip_existing_file(log)
 
     if build_complete:
-
+        # must ensure there's an lid
+        lid = os.environ["LID"] if "LID" in os.environ else get_timestamp("%y%m%d-%H%M%S")
+        if save_build_provenance:
+            save_build_provenance_sub(case, lid=lid)
         # Set XML to indicate build complete
         case.set_value("BUILD_COMPLETE", True)
         case.set_value("BUILD_STATUS", 0)
@@ -556,14 +561,12 @@ def post_build(case, logs, build_complete=False):
 
         lock_file("env_build.xml")
 
-        # must ensure there's an lid
-        lid = os.environ["LID"] if "LID" in os.environ else get_timestamp("%y%m%d-%H%M%S")
-        save_build_provenance(case, lid=lid)
 
 ###############################################################################
-def case_build(caseroot, case, sharedlib_only=False, model_only=False, buildlist=None):
+def case_build(caseroot, case, sharedlib_only=False, model_only=False, buildlist=None, save_build_provenance=True):
 ###############################################################################
-    functor = lambda: _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist)
+    functor = lambda: _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist,
+                                       save_build_provenance)
     return run_and_log_case_status(functor, "case.build", caseroot=caseroot)
 
 ###############################################################################
diff --git a/scripts/lib/CIME/provenance.py b/scripts/lib/CIME/provenance.py
--- a/scripts/lib/CIME/provenance.py
+++ b/scripts/lib/CIME/provenance.py
@@ -5,7 +5,7 @@
 """
 
 from CIME.XML.standard_module_setup import *
-from CIME.utils import touch, gzip_existing_file, SharedArea, copy_umask, convert_to_babylonian_time, get_current_commit
+from CIME.utils import touch, gzip_existing_file, SharedArea, copy_umask, convert_to_babylonian_time, get_current_commit, indent_string, run_cmd, run_cmd_no_fail
 
 import tarfile, getpass, signal, glob, shutil, sys
 
@@ -80,10 +80,30 @@ def _save_build_provenance_cesm(case, lid): # pylint: disable=unused-argument
     version = case.get_value("MODEL_VERSION")
     # version has already been recorded
     srcroot = case.get_value("SRCROOT")
-    manic = os.path.join(srcroot, "manage_externals","checkout_externals")
+    manic = os.path.join("manage_externals","checkout_externals")
+    manic_full_path = os.path.join(srcroot, manic)
     out = None
-    if os.path.exists(manic):
-        out = run_cmd_no_fail(manic + " --status --verbose --no-logging", from_dir=srcroot)
+    if os.path.exists(manic_full_path):
+        args = " --status --verbose --no-logging"
+        stat, out, err = run_cmd(manic_full_path + args, from_dir=srcroot)
+        errmsg = """Error gathering provenance information from manage_externals.
+
+manage_externals error message:
+{err}
+
+manage_externals output:
+{out}
+
+To solve this, either:
+
+(1) Find and fix the problem: From {srcroot}, try to get this command to work:
+    {manic}{args}
+
+(2) If you don't need provenance information, rebuild with --skip-provenance-check
+""".format(out=indent_string(out, 4), err=indent_string(err, 4),
+           srcroot=srcroot, manic=manic, args=args)
+        expect(stat==0,errmsg)
+
     caseroot = case.get_value("CASEROOT")
     with open(os.path.join(caseroot, "CaseStatus"), "a") as fd:
         if version is not None and version != "unknown":
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -1563,6 +1563,24 @@ def stringify_bool(val):
     expect(type(val) is bool, "Wrong type for val '{}'".format(repr(val)))
     return "TRUE" if val else "FALSE"
 
+def indent_string(the_string, indent_level):
+    """Indents the given string by a given number of spaces
+
+    Args:
+       the_string: str
+       indent_level: int
+
+    Returns a new string that is the same as the_string, except that
+    each line is indented by 'indent_level' spaces.
+
+    In python3, this can be done with textwrap.indent.
+    """
+
+    lines = the_string.splitlines(True)
+    padding = ' ' * indent_level
+    lines_indented = [padding + line for line in lines]
+    return ''.join(lines_indented)
+
 def verbatim_success_msg(return_val):
     return return_val
 


Test patch:
diff --git a/scripts/lib/CIME/SystemTests/system_tests_common.py b/scripts/lib/CIME/SystemTests/system_tests_common.py
--- a/scripts/lib/CIME/SystemTests/system_tests_common.py
+++ b/scripts/lib/CIME/SystemTests/system_tests_common.py
@@ -119,8 +119,10 @@ def build_indv(self, sharedlib_only=False, model_only=False):
         """
         Perform an individual build
         """
+        model = self._case.get_value('MODEL')
         build.case_build(self._caseroot, case=self._case,
-                         sharedlib_only=sharedlib_only, model_only=model_only)
+                         sharedlib_only=sharedlib_only, model_only=model_only,
+                         save_build_provenance=not model=='cesm')
 
     def clean_build(self, comps=None):
         if comps is None:
diff --git a/scripts/lib/CIME/tests/test_utils.py b/scripts/lib/CIME/tests/test_utils.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/CIME/tests/test_utils.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+
+import unittest
+from CIME.utils import indent_string
+
+class TestIndentStr(unittest.TestCase):
+    """Test the indent_string function.
+
+    """
+
+    def test_indent_string_singleline(self):
+        """Test the indent_string function with a single-line string
+
+        """
+        mystr = 'foo'
+        result = indent_string(mystr, 4)
+        expected = '    foo'
+        self.assertEqual(expected, result)
+
+    def test_indent_string_multiline(self):
+        """Test the indent_string function with a multi-line string
+
+        """
+        mystr = """hello
+hi
+goodbye
+"""
+        result = indent_string(mystr, 2)
+        expected = """  hello
+  hi
+  goodbye
+"""
+        self.assertEqual(expected, result)
+
+if __name__ == '__main__':
+    unittest.main()
+