Repository: ESMCI/cime

Issue:
Record number of nodes for given job
Presently, CIME computes the number of nodes required for a job submission but this information is not recorded anywhere AFAIK. Can we store it for archiving purposes?


Discussion:
@sarats , so you want to add this to the E3SM provenance?
Yes eventually.  Ideally it would be recorded in an xml variable.  But if it was output as part of ./preview_run (on its own line) that would help.
Note, there is no reason to make this ONLY apply to E3SM. It should apply to all models, there's no reason to make it work only for one. And actually making it only work for one adds more work, so you might as well make it easier on yourself and have it work for all.
We weren't implying this should only be done for E3SM.  It should be general CIME CCS feature.  But E3SM's provenance recording system will process it a certain way.
I think that this is already recorded for a given run in several places.  It's in the log files,
and it's in the timing files.  I think introducing another xml variable for this is unwarranted. 
If you really want this information for provenance then you should save the env_mach_pes.xml
Edit/Update: Never mind, had to refresh to get latest comments. We already capture env_mach_pes.xml etc. The timing summary (e3sm_timing.<case>) & related files do not record num_nodes. 

_Yes, but probably CESM would like it too?_ 

_In fact, it might be capture 'preview_run.log' as well if not already? That could contain the mpirun/jsrun/srun invocation as well which could be handy along with the `replay.sh` script._ 

Here's the output of preview_run from a recent case I ran:
```
[jacob@chrlogin1 Ftestrh]$  ./preview_run 
CASE INFO:
  nodes: 3
  total tasks: 96
  tasks per node: 32
  thread count: 2

```
The value of 3 for "nodes" doesn't appear in any normal XML files or logs.    The hidden .case.run files has "#SBATCH  --nodes=3" at the top but that's an odd place to look.
To add, the directives in .case.run would be dependent on the batch system on a given platform, so it would prudent to have a more generic output. 

Additionally, our mpirun/srun/jsrun invocations often use runtime logic (inline shell commands) to construct the final command-line. So, we should also capture preview_run.log while we're at this.
@rljacob @sarats @jedwards4b 
Would there be any benefit to store this in a machine readable format? Or do we want to stick with output similar to `./preview_run`?

Lastly, just want confirm that details in `./preview_run` are sufficient?
@jasonb5 Can you confirm where the number of nodes information is stored? Is that archived in provenance.py?
I see it in preview_run.log.* in the CASEDIR.  At the top, that file has:

```
CASE INFO:
  nodes: 48
  total tasks: 1728
  tasks per node: 36
  thread count: 1
  ngpus per node: 0
```
So, preview_run.log is being archived (https://github.com/ESMCI/cime/blob/231c8e783d9df1896f7ac0bdd1f31e74f496bdac/scripts/lib/CIME/provenance.py#L345).

However, I don't see it for recent water cycle experiments (Version: v2.0.0-beta.3-17386-ge3472d21b2). Perhaps they need to pull in a CIME update?
We probably need to get the fix for replay.sh in their update as well along with GIT_REMOTE and GIT_CONFIG commits.
They're stored in run directory only.

I'll need to make another fix to have them added to the performance archive.

Should all the GIT_* be included in the perforamance archive as well?
Yes, all of these new features you added recently - those files should be saved in the archive.
I'll make a PR to address this.

Resolving patch:
diff --git a/scripts/lib/CIME/provenance.py b/scripts/lib/CIME/provenance.py
--- a/scripts/lib/CIME/provenance.py
+++ b/scripts/lib/CIME/provenance.py
@@ -347,11 +347,18 @@ def _save_prerun_provenance_common(case, lid):
     """
     run_dir = case.get_value("RUNDIR")
 
-    preview_log = os.path.join(run_dir, "preview_run.log.{}".format(lid))
+    base_preview_run = os.path.join(run_dir, "preview_run.log")
+    preview_run = f"{base_preview_run}.{lid}"
 
-    with open(preview_log, "w") as fd:
+    if os.path.exists(base_preview_run):
+        os.remove(base_preview_run)
+
+    with open(base_preview_run, "w") as fd:
         case.preview_run(lambda x: fd.write("{}\n".format(x)), None)
 
+        # Create copy rather than symlink, the log is automatically gzipped
+        safe_copy(base_preview_run, preview_run)
+
 def save_prerun_provenance(case, lid=None):
     with SharedArea():
         # Always save env
@@ -474,6 +481,12 @@ def _save_postrun_timing_e3sm(case, lid):
     globs_to_copy.append("timing/*.{}*".format(lid))
     globs_to_copy.append("CaseStatus")
     globs_to_copy.append(os.path.join(rundir, "spio_stats.{}.tar.gz".format(lid)))
+    # Can't use a single glob, similar files e.g. {filename}.{lid} get picked up.
+    bld_filenames = ["GIT_STATUS", "GIT_DIFF", "GIT_LOG", "GIT_REMOTE",
+                     "GIT_CONFIG", "GIT_SUBMODULE_STATUS"]
+    bld_globs = map(lambda x: f"bld/{x}", bld_filenames)
+    globs_to_copy.extend(bld_globs)
+    globs_to_copy.append("run/preview_run.log")
 
     for glob_to_copy in globs_to_copy:
         for item in glob.glob(os.path.join(caseroot, glob_to_copy)):


Test patch: