Repository: ESMCI/cime

Issue:
Correct the multi-driver coupler history file names
I'm running a multi-driver CAM6 and asking for coupler history files.
The files are being generated with the instance number in the wrong
place in the file names:
CAM6_resource80.cpl.ha2x3h_0001.2010-07-01.nc
instead of 
CAM6_resource80.cpl_0001.ha2x3h.2010-07-01.nc

I've found where this is happening
(seq_hist_writeaux in cime_comp_mod.F90 and seq_hist_mod.F90)
so I can fix this and issue a PR after someone tells me which
CIME version I should clone.  Last time, when I tried to update
my (kdraeder) github cime master from the ESMCI master, 
I found out that it was the wrong one.  
I was supposed to update from something like maint-#.#.
Please point me to the right version.
Thanks,
Kevin


Discussion:
If your changes are appropriate for cesm2.0.x or cesm2.1.x the changes should go in the maint-5.6 branch otherwise they should go in master.   
I've installed a fix into cesm2_2_alpha01a + maint-5.6:
src/drivers/mct/main/cime_comp_mod.F90
src/drivers/mct/main/seq_hist_mod.F90
to attach the instance number to 'cpl' instead of aname.
It works in my test case.

I ran scripts_regression_tests.py and see some failures:
FAIL: test_save_timings (__main__.L_TestSaveTimings)
FAIL: test_save_timings_manual (__main__.L_TestSaveTimings)
FAIL: test_b_full (__main__.O_TestTestScheduler)
FAIL: test_c_use_existing (__main__.O_TestTestScheduler)
FAIL: test_d_retry (__main__.O_TestTestScheduler)
        ERROR: BUILD FAIL: Intentional fail for testing infrastructure
FAIL: test_bless_test_results (__main__.Q_TestBlessTestResults)
FAIL: test_run_restart (__main__.T_TestRunRestart)
FAIL: test_run_restart_too_many_fails (__main__.T_TestRunRestart)
FAIL: test_full_system (__main__.Z_FullSystemTest)

Do I need to look into any of these?
The CESM is in /gpfs/fs1/work/raeder/Models/cesm2_0_maint-5.6
All of these tests are expected to pass. But If your changes are intended for cesm2.2 you should put them on master not the maint-5.6 branch.  Please reread my comment above. 
I didn't intend them for 2.2, partly because I wasn't aware of its existence.
I thought that 2.1 was the latest, since the next CESM release is expected
to be 2.1.  So it seemed appropriate to get the CESM default and plug in
maint-5.6, which I thought would be following your comment.  When I was 
done fixing and testing I learned that it was 2.2.

I don't have an intended version number.
I want to implement my fixes in the CIME that is most appropriate
in the current development environment.  I want a CESM that
works well with that CIME, so that any testing failures are the
result of my coding mistakes, rather than CIME-CESM  version mismatches.
There are ~25 cesm2.0 and 2.1 tags visible from '> git tag', 
so it seems like asking is the only practical way for me to choose one.
 
Since I didn't know that I should ask about a CESM version that goes
with maint-5.6, I guess I should ask "Are there other pieces of software
that I need to choose to play well with this CIME and CESM?"
Okay, maint-5.6 is where they belong then.  SInce the scripts_regression_tests.py test only cime and do not depend on the cesm tag I would still expect all of those tests to pass.    
I restored the original versions of cime_comp_mod.F90
and seq_hist_mod.F90, and reran scripts_regression_test.py,
still with cesm2_2_alpha01a.  I got the same error messages.
So I should try again with a different CESM.  If you can recommend
one, I'll use that.  Otherwise I'll guess and hope for the best.

The cesm version has nothing to do with the test failures you are seeing.  If you would like to stop by I can try to help you debug.
@jedwards4b
Thanks for pointing out that I was confusing myself by
using this branch for too many developments and commenting
out the lines in case_run.py that run the model.

I believe that I finally figured out a way 
to import maint-5.6 into cesm2_2_alpha01a.
It's in /gpfs/fs1/work/raeder/Models/cesm2_2_maint-5.6.
That combination does pass scripts_regression_tests.py.

When I install modified cime_comp_mod.F90 and seq_hist_mod.F90
to fix the multi-instance cpl history file names
it fails only TESTMEMLEAKFAIL_P1.f09_g16.X.cheyenne_intel.

An error message 
"ERROR: Model did not complete - see 
 /glade/scratch/raeder/scripts_regression_test.20180823_223401/TESTMEMLEAKFAIL_P1.f09_g16.X.cheyenne_intel.fake_testing_only_20180823_225258-20180823_225258/run/**cpl.log**.1767443.chadmin1.180823-225408"
is printed from case_run.py line 198, which,
I think, is only printed if that cpl.log file is missing,
so it's not a helpful error message.

Even if that file existed, I believe that the script would fail there
because it goes on to search for "SUCCESSFUL TERMINATION"
in the cpl log file, but that string is not echoed by
TESTMEMLEAKFAIL_P1.f09_g16.X.cheyenne_intel.fake_testing_only_20180823_225258-20180823_225258/bld/**cesm.exe**
like it is for other tests (TESTRUNSLOWPASS, ...).
If it were echoed, the cpl log file might exist.
(I haven't figured out the relationship between the cesm.exe script 
and the TESTMEMLEAKFAIL test in system_tests_common.py:~635, 
which also is "missing" an echo of "SUCCESSFUL TERMINATION"
in the """...""" section.)

The string 'Insta pass' is printed to
TESTMEMLEAKFAIL_P1.f09_g16.X.cheyenne_intel.fake_testing_only_20180823_225258-20180823_225258/run/**cesm.log**.1767443.chadmin1.180823-225408
so cesm.exe seems to be behaving as created.

When I insert a line to echo success to the cpl log file,
it seems to finish, but there could be a lurking problem.
I tested with 
scripts_regression_tests.py O_TestTestScheduler.test_b_full.
There's still a failure in test_b_full, but now it's because the
test status is PASS instead of the expected FAIL.
In previous tests it was None, instead of FAIL.
Maybe I missed the point of the test, to intentionally fail?
There was no 
/gpfs/fs1/work/raeder/Models/cesm2_2_maint-5.6/cime/scripts/tests/cpl.log.failmemleak.gz
to gunzip into the cpl log file in $rundir.
See /gpfs/fs1/scratch/raeder/scripts_regression_test.20180824_133156/TESTMEMLEAKFAIL_P1.f09_g16.X.cheyenne_intel.fake_testing_only_20180824_133157-20180824_133157

If this looks like a bug, it should probably be handled
in a separate issue.

So I think you are ready to submit a pull request for review.
@jedwards4b Should it include the 'fixes' for the MEMLEAKFAIL test?

Resolving patch:
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -754,7 +754,7 @@ def _setup_mach_pes(self, pecount, multi_driver, ninst, machine_name, mpilib):
 
     def configure(self, compset_name, grid_name, machine_name=None,
                   project=None, pecount=None, compiler=None, mpilib=None,
-                  pesfile=None,user_grid=False, gridfile=None,
+                  pesfile=None, gridfile=None,
                   multi_driver=False, ninst=1, test=False,
                   walltime=None, queue=None, output_root=None,
                   run_unsupported=False, answer=None,
@@ -773,11 +773,10 @@ def configure(self, compset_name, grid_name, machine_name=None,
             compset_name, files)
 
         self._components = self.get_compset_components()
+
         #--------------------------------------------
         # grid
         #--------------------------------------------
-        if user_grid is True and gridfile is not None:
-            self.set_value("GRIDS_SPEC_FILE", gridfile)
         grids = Grids(gridfile)
 
         gridinfo = grids.get_grid_info(name=grid_name, compset=self._compsetname)
@@ -1421,7 +1420,7 @@ def get_latest_cpl_log(self, coupler_log_path=None):
     def create(self, casename, srcroot, compset_name, grid_name,
                user_mods_dir=None, machine_name=None,
                project=None, pecount=None, compiler=None, mpilib=None,
-               pesfile=None,user_grid=False, gridfile=None,
+               pesfile=None, gridfile=None,
                multi_driver=False, ninst=1, test=False,
                walltime=None, queue=None, output_root=None,
                run_unsupported=False, answer=None,
@@ -1436,7 +1435,7 @@ def create(self, casename, srcroot, compset_name, grid_name,
             self.configure(compset_name, grid_name, machine_name=machine_name,
                            project=project,
                            pecount=pecount, compiler=compiler, mpilib=mpilib,
-                           pesfile=pesfile,user_grid=user_grid, gridfile=gridfile,
+                           pesfile=pesfile, gridfile=gridfile,
                            multi_driver=multi_driver, ninst=ninst, test=test,
                            walltime=walltime, queue=queue,
                            output_root=output_root,


Test patch: