Repository: ESMCI/cime

Issue:
Staging a hybrid run is broken
When I create a hybrid run using the following commands:
 >./xmlchange RUN_REFCASE=b.e11.B1850C5CN.f09_g16.005_v3                           
 ./xmlchange RUN_REFDATE=0402-01-01                                               
 ./xmlchange GET_REFCASE=TRUE   
 ./xmlchange RUN_TYPE=hybrid

I get the following message from preview_namelists:
>Creating component namelists
   Calling cam buildnml
     ...calling cam buildcpp to set build time options
   Calling clm buildnml
   Calling cice buildnml
     ...calling cice buildcpp to set build time options
   Calling pop buildnml
     ...calling pop buildcpp to set build time options
Traceback (most recent call last):
  File "./preview_namelists", line 48, in <module>
    _main_func(__doc__)
  File "./preview_namelists", line 45, in _main_func
    create_namelists(case, component=args.component)
  File "/glade2/scratch2/cacraig/cam5_4_138_trunk06/cime/scripts/Tools/../../scripts/lib/CIME/preview_namelists.py", line 87, in create_namelists
    mod.buildnml(case, caseroot, compname)
  File "/glade2/scratch2/cacraig/cam5_4_138_trunk06/components/pop/cime_config/buildnml", line 149, in buildnml
    if 'RESTART_FMT=nc' in open(pointer_file).read():
IOError: [Errno 2] No such file or directory: '/glade/scratch/cacraig/test_cecile_b1850_trunk06/run/rpointer.ocn.restart'

My workaround is to hand copy the rpointer.ocn.restart file and the file it references into my RUNDIR.  Since I am building the B1850  compset, this is not a viable long-term solution.

From an email exchange with Mike Levy, he had the following reply:
>Hi Cheryl,

>Do you think the run doesn't need the refcase documents, or do you think CESM should handle the copy for you?

>I actually ran into this exact error with a grad student yesterday, and convinced myself that manually copying the refcase files _was_ necessary for a hybrid because (unlike a branch) you may want to initialize one (or more) of the components as a startup instead of from a restart. Looking closer, though, I'm not sure that first case (starting POP from initial condition files) would build out of the box... this logic in $POP/cime_config/buildnml is triggering the error you are seeing:

        if run_type == 'startup' and not continue_run:
            check_pointer_file = False
        else:
            check_pointer_file = True

        format = 'bin'
        if check_pointer_file:
            if 'RESTART_FMT=nc' in open(pointer_file).read():

>If you want to do a hybrid run and start POP from initial condition files, then check_pointer_file should be False but the scripts apparently don't support that.

>~Mike




Discussion:
Could you please run check_input_data on your hybrid case with the --debug flag.   
yslogin2$ ./check_input_data --debug
RUN: /glade/apps/opt/usr/bin/xmllint --noout --schema /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/config/xml_schemas/env_entry_id.xsd /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_case.xml
  errput: /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_case.xml validates

RUN: /glade/apps/opt/usr/bin/xmllint --noout --schema /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/config/xml_schemas/env_entry_id.xsd /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_run.xml
  errput: /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_run.xml validates

RUN: /glade/apps/opt/usr/bin/xmllint --noout --schema /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/config/xml_schemas/env_entry_id.xsd /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_build.xml
  errput: /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_build.xml validates

RUN: /glade/apps/opt/usr/bin/xmllint --noout --schema /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/config/xml_schemas/env_mach_pes.xsd /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_mach_pes.xml
  errput: /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_mach_pes.xml validates

RUN: /glade/apps/opt/usr/bin/xmllint --noout --schema /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/config/xml_schemas/env_batch.xsd /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_batch.xml
  errput: /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_batch.xml validates

RUN: /glade/apps/opt/usr/bin/xmllint --noout --schema /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/config/xml_schemas/env_mach_specific.xsd /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_mach_specific.xml
  errput: /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_mach_specific.xml validates

RUN: /glade/apps/opt/usr/bin/xmllint --noout --schema /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/config/xml_schemas/env_archive.xsd /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_archive.xml
  errput: /glade2/scratch2/cacraig/cam5_4_138_trunk05/cime/scripts/for_jim/env_archive.xml validates

Loading input file list: 'Buildconf/clm.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'

The problem is that check_input_data is called by case.submit but the pop buildnml expects the rpointer file to be in the run directory when preview_namelists is run. I think that we need to add a call to check_input_data in the preview_namelist script if get_refcase is true.

Resolving patch:
diff --git a/scripts/lib/CIME/check_input_data.py b/scripts/lib/CIME/check_input_data.py
--- a/scripts/lib/CIME/check_input_data.py
+++ b/scripts/lib/CIME/check_input_data.py
@@ -73,6 +73,9 @@ def check_all_input_data(case):
     success = check_input_data(case=case, download=True)
     expect(success, "Failed to download input data")
 
+    stage_refcase(case)
+
+def stage_refcase(case):
     get_refcase  = case.get_value("GET_REFCASE")
     run_type     = case.get_value("RUN_TYPE")
     continue_run = case.get_value("CONTINUE_RUN")
@@ -113,17 +116,18 @@ def check_all_input_data(case):
             logger.debug("Creating run directory: {}".format(rundir))
             os.makedirs(rundir)
 
-        for rcfile in glob.iglob(os.path.join(refdir,"*{}*".format(run_refcase))):
-            logger.debug("Staging file {}".format(rcfile))
-            rcbaseline = os.path.basename(rcfile)
-            if not os.path.exists("{}/{}".format(rundir, rcbaseline)):
-                os.symlink(rcfile, "{}/{}".format(rundir, rcbaseline))
-
         # copy the refcases' rpointer files to the run directory
-        for rpointerfile in  glob.iglob(os.path.join("{}","*rpointer*").format(refdir)):
+        for rpointerfile in glob.iglob(os.path.join("{}","*rpointer*").format(refdir)):
             logger.debug("Copy rpointer {}".format(rpointerfile))
             shutil.copy(rpointerfile, rundir)
 
+        # link everything else
+
+        for rcfile in glob.iglob(os.path.join(refdir,"*")):
+            rcbaseline = os.path.basename(rcfile)
+            if not os.path.exists("{}/{}".format(rundir, rcbaseline)):
+                logger.debug("Staging file {}".format(rcfile))
+                os.symlink(rcfile, "{}/{}".format(rundir, rcbaseline))
 
         for cam2file in  glob.iglob(os.path.join("{}","*.cam2.*").format(rundir)):
             camfile = cam2file.replace("cam2", "cam")
diff --git a/scripts/lib/CIME/preview_namelists.py b/scripts/lib/CIME/preview_namelists.py
--- a/scripts/lib/CIME/preview_namelists.py
+++ b/scripts/lib/CIME/preview_namelists.py
@@ -4,6 +4,7 @@
 
 from CIME.XML.standard_module_setup import *
 from CIME.utils import run_sub_or_cmd
+from CIME.check_input_data import stage_refcase
 import glob, shutil
 logger = logging.getLogger(__name__)
 
@@ -57,6 +58,9 @@ def create_namelists(case, component=None):
     # Load modules
     case.load_env()
 
+    stage_refcase(case)
+
+
     logger.info("Creating component namelists")
 
     # Create namelists - must have cpl last in the list below


Test patch: