Repository: ESMCI/cime

Issue:
Consider upgrading status of the job ID
When submitted jobs to a batch system, the job ID is a critical integer and could be made more available.  One place it seems it could easily be added is in the CaseStatus.

```
2017-11-29 14:01:27: case.submit starting 
 ---------------------------------------------------
2017-11-29 14:02:15: case.submit success  JOB ID is 42
 ---------------------------------------------------
2017-11-29 15:37:35: model execution success 
```


Jim F. suggested a query that doesn't do what I would expect: 
```
cori08% pwd
/global/cscratch1/sd/ndk/ACME_simulations/2017-master-nov28.hmod526t03.mnov28.st04.atune.long.ne120np4_oRRS18to6v3_ICG.cori-knl/case_scripts
cori08% xmlquery JOB_IDS
        JOB_IDS: case.run:8787607
cori08% 
```

The job ID returned is the ID for the job that is currently submitted, but I'd like to know the job ID of the previous job that just failed (for example).

I might even argue to put the job ID with the date/time stamp for the log files.  `cpl.log.171129-022802` could be `cpl.log.171129-022802.8787607`.

Also, after the job is complete, the slurm system (at least) has a nice way of getting some info from the job:
```
cori08% sacct -j 8700613 -X --format=JobID,JobName,NNodes,Elapsed,Account,QOS,State
       JobID    JobName   NNodes    Elapsed    Account        QOS      State 
------------ ---------- -------- ---------- ---------- ---------- ---------- 
8700613      hmod600t0+      600   01:23:35       acme normal_kn+     FAILED 

or:

cori08% sacct -j 8700613 -a -o JobID%-20,JobName,State,ExitCode,DerivedExitCode,Elapsed,NNodes,NTasks,MaxRSS
               JobID    JobName      State ExitCode DerivedExitCode    Elapsed   NNodes   NTasks     MaxRSS 
-------------------- ---------- ---------- -------- --------------- ---------- -------- -------- ---------- 
8700613              hmod600t0+     FAILED      1:0             0:0   01:23:35      600                     
8700613.batch             batch     FAILED      1:0                   01:23:35        1        1     43568K 
8700613.extern           extern  COMPLETED      0:0                   01:25:03      600      600          0 
8700613.0              acme.exe CANCELLED+      0:9                   00:25:13      600    38400    548784K 

```

This will tell you exactly how much time the entire job used and if it failed.  The memory reported in the second command is only moderately useful, but can point to a job that used too much memory.



Discussion:
@jedwards4b any thoughts on this? At first glance, adding tis info to CaseStatus seemed the easiest approach.
Sending it to CaseStatus seems entirely appropriate.   We can make the addition of jobid to the log files an option too.   Probably we should have a string format option for the log file names.
OK, I am going to consider the discussion over and convert this to action ticket.

Resolving patch:
diff --git a/scripts/lib/CIME/case_submit.py b/scripts/lib/CIME/case_submit.py
--- a/scripts/lib/CIME/case_submit.py
+++ b/scripts/lib/CIME/case_submit.py
@@ -7,7 +7,7 @@
 """
 import socket
 from CIME.XML.standard_module_setup import *
-from CIME.utils                     import expect, run_and_log_case_status
+from CIME.utils                     import expect, run_and_log_case_status, verbatim_success_msg
 from CIME.preview_namelists         import create_namelists
 from CIME.check_lockedfiles         import check_lockedfiles
 from CIME.check_input_data          import check_all_input_data
@@ -82,6 +82,8 @@ def _submit(case, job=None, no_batch=False, prereq=None, resubmit=False,
     if xml_jobid_text:
         case.set_value("JOB_IDS", xml_jobid_text)
 
+    return xml_jobid_text
+
 def submit(case, job=None, no_batch=False, prereq=None, resubmit=False,
            skip_pnl=False, mail_user=None, mail_type=None, batch_args=None):
     if case.get_value("TEST"):
@@ -102,7 +104,8 @@ def submit(case, job=None, no_batch=False, prereq=None, resubmit=False,
                                   resubmit=resubmit, skip_pnl=skip_pnl,
                                   mail_user=mail_user, mail_type=mail_type,
                                   batch_args=batch_args)
-        run_and_log_case_status(functor, "case.submit", caseroot=case.get_value("CASEROOT"))
+        run_and_log_case_status(functor, "case.submit", caseroot=case.get_value("CASEROOT"),
+                                custom_success_msg_functor=verbatim_success_msg)
     except:
         # If something failed in the batch system, make sure to mark
         # the test as failed if we are running a test.
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -1420,7 +1420,10 @@ def stringify_bool(val):
     expect(type(val) is bool, "Wrong type for val '{}'".format(repr(val)))
     return "TRUE" if val else "FALSE"
 
-def run_and_log_case_status(func, phase, caseroot='.'):
+def verbatim_success_msg(return_val):
+    return return_val
+
+def run_and_log_case_status(func, phase, caseroot='.', custom_success_msg_functor=None):
     append_case_status(phase, "starting", caseroot=caseroot)
     rv = None
     try:
@@ -1430,7 +1433,8 @@ def run_and_log_case_status(func, phase, caseroot='.'):
         append_case_status(phase, "error", msg=("\n{}".format(e)), caseroot=caseroot)
         raise
     else:
-        append_case_status(phase, "success", caseroot=caseroot)
+        custom_success_msg = custom_success_msg_functor(rv) if custom_success_msg_functor else None
+        append_case_status(phase, "success", msg=custom_success_msg, caseroot=caseroot)
 
     return rv
 


Test patch: