Repository: ESMCI/cime

Issue:
Printenv should be part of case.run
Something like `os.system('printenv >& run_env.txt')`; always on by default. This would help in debugging issues with env-vars on compute nodes.


Discussion:
@amametjanov software_environment.txt should meet this need, right?
I thought so too, but it appears that software_environment.txt is generated on login nodes. So run-time env-vars like `PBS_O_PATH` are missing in that file.
Ah, good point. So maybe we need to either delay or overwrite software_environment.txt from the compute nodes.
software_environment.txt is meant primarily to capture the build-time environment. As long as it is not overwritten until it has been copied to the performance archive, this will not bother me personally, but not everyone will be using the performance archive. Having both build and runtime environments captured is probably a better solution. (If we do capture both, we'll want to copy both to the performance archvie.)
Yes, lets rename software_environment.txt to build_environment.txt and also output a run_environment.txt.
OK, I will do that.
I think `run_environment.YYMMDD-HHMMSS.txt` should also be placed in the run-dir so that there is a record of how the environment changed between two or more runs.

BTW, need some cleanup in the bld-dir:
- remove sym-links from GIT_DESCRIBE, SourceMods.tar.gz, build_environment.txt to the latest time-stamped versions. All three files are time-stamped already.
- remove comp.bldlog.time-stamp files after gzip

Putting these environment files into the run dir is really going to clutter it up when doing long production runs - how about putting them into the log subdirectory instead (with the same LID as the model logs have)
Sounds good. I think the three bld-dir files (GIT_DESCRIBE, SourceMods.tar.gz, build_environment.txt) should be placed into `provenance` subdirectory to also reduce clutter.

Resolving patch:
diff --git a/utils/python/CIME/XML/env_mach_specific.py b/utils/python/CIME/XML/env_mach_specific.py
--- a/utils/python/CIME/XML/env_mach_specific.py
+++ b/utils/python/CIME/XML/env_mach_specific.py
@@ -119,6 +119,15 @@ def list_modules(self):
         else:
             expect(False, "Unhandled module system '%s'" % module_system)
 
+    def save_all_env_info(self, filename):
+        """
+        Get a string representation of all current environment info and
+        save it to file.
+        """
+        with open(filename, "w") as f:
+            f.write(self.list_modules())
+        run_cmd_no_fail("echo -e '\n' >> %s && env >> %s" % (filename, filename))
+
     def make_env_mach_specific_file(self, compiler, debug, mpilib, shell):
         modules_to_load = self._get_modules_for_case(compiler, debug, mpilib)
         envs_to_set = self._get_envs_for_case(compiler, debug, mpilib)
diff --git a/utils/python/CIME/case_setup.py b/utils/python/CIME/case_setup.py
--- a/utils/python/CIME/case_setup.py
+++ b/utils/python/CIME/case_setup.py
@@ -264,10 +264,7 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
         env_module = case.get_env("mach_specific")
         env_module.make_env_mach_specific_file(compiler, debug, mpilib, "sh")
         env_module.make_env_mach_specific_file(compiler, debug, mpilib, "csh")
-        with open("software_environment.txt", "w") as f:
-            f.write(env_module.list_modules())
-        run_cmd_no_fail("echo -e '\n' >> software_environment.txt && \
-                         env >> software_environment.txt")
+        env_module.save_all_env_info("software_environment.txt")
 
 ###############################################################################
 def case_setup(case, clean=False, test_mode=False, reset=False, no_status=False):
diff --git a/utils/python/CIME/provenance.py b/utils/python/CIME/provenance.py
--- a/utils/python/CIME/provenance.py
+++ b/utils/python/CIME/provenance.py
@@ -193,6 +193,10 @@ def save_prerun_provenance_cesm(case, lid=None): # pylint: disable=unused-argume
 
 def save_prerun_provenance(case, lid=None):
     with SharedArea():
+        # Always save env
+        lid = os.environ["LID"] if lid is None else lid
+        env_module = case.get_env("mach_specific")
+        env_module.save_all_env_info(os.path.join(case.get_value("CASEROOT"), "logs", "run_environment.txt.%s" % lid))
         model = case.get_value("MODEL")
         if model == "acme":
             save_prerun_provenance_acme(case, lid=lid)
@@ -253,6 +257,7 @@ def save_postrun_provenance_acme(case, lid):
     elif mach in ["edison", "cori-haswell", "cori-knl"]:
         globs_to_copy.append("%s" % case.get_value("CASE"))
 
+    globs_to_copy.append("logs/run_environment.txt.%s" % lid)
     globs_to_copy.append("logs/acme.log.%s.gz" % lid)
     globs_to_copy.append("logs/cpl.log.%s.gz" % lid)
     globs_to_copy.append("timing/*.%s" % lid)


Test patch: