Repository: ESMCI/cime

Issue:
Branch a single instance case from a multi-instance case
When a multi-instance CAM forecast fails, I want to use one of the failed instances
as an exact restart in a single instance case, to speed debugging.
Building it as hybrid or startup won't work because for CAM 
those are not exact restarts; they use the CAM initial file
I first tried building a single instance branch run with the multi-instance
as the refcase.   That branched the entire multi-instance run.

Then I tried copying all of the restart files for 1 instance into a new directory, 
"Restarts", with instance numbers removed from the file names.
I built the single instance case with Restarts as the RUN_REFDIR.
It built, but when I tried to run it it complained about a mozart file
already existing in the run directory.  I believe that an earlier stage
of the submit process created that file, so I'm stumped about how
to fix this problem.  I've played with GET_REFCASE and CONTINUE_RUN
and other things, but have not found the magic combination.
Am I missing something, or is this a new capability that hasn't been implemented?

$CASEROOT = /gpfs/fs1/work/raeder/Exp/Debug_lwdn4
$RUNDIR = /gpfs/fs1/scratch/raeder/Debug_lwdn4/run
$CESMROOT = /glade/work/raeder/Models/cesm2_2_maint-5.6

Thanks
Kevin



Discussion:
Hold on.  I may have figured it out.  More testing, then I'll report.
The following code contains what I believe are differences
from building a plain branch run, which enable building 
a single instance branch run from a multi-instance reference case.
There may be extraneous bits in it, but I don't have time to trim it right now.
```
setenv stagedir /glade/scratch/${USER}/${refcase}/run
setenv rundir   /glade/scratch/${USER}/${case}/run

./xmlchange RUN_REFDIR=$rundir
./xmlchange GET_REFCASE=FALSE

set init_time = ${RUN_REFDATE}-${RUN_REFTOD}
set inst = 3
set inst_string = `printf _%04d $inst`
ln -sf ${stagedir}/${refcase}.cam${inst_string}.r.${init_time}.nc      ${refcase}.cam.r.${init_time}.nc
ln -sf ${stagedir}/${refcase}.cam${inst_string}.rs.${init_time}.nc     ${refcase}.cam.rs.${init_time}.nc
ln -sf ${stagedir}/${refcase}.clm2${inst_string}.r.${init_time}.nc     ${refcase}.clm2.r.${init_time}.nc
ln -sf ${stagedir}/${refcase}.clm2${inst_string}.rh0.${init_time}.nc   ${refcase}.clm2.rh0.${init_time}.nc
ln -sf ${stagedir}/${refcase}.cice${inst_string}.r.${init_time}.nc     ${refcase}.cice.r.${init_time}.nc
ln -sf ${stagedir}/${refcase}.cpl${inst_string}.r.${init_time}.nc      ${refcase}.cpl.r.${init_time}.nc
ln -sf ${stagedir}/${refcase}.docn${inst_string}.rs1.${init_time}.bin  ${refcase}.docn.rs1.${init_time}.bin
echo "${refcase}.cam.r.${init_time}.nc"     >! rpointer.atm
echo "${refcase}.clm2.r.${init_time}.nc"    >! rpointer.lnd
echo "${refcase}.cice.r.${init_time}.nc"    >! rpointer.ice
echo "${refcase}.cpl.r.${init_time}.nc"     >! rpointer.drv
echo "${refcase}.docn.r.${init_time}.nc"    >! rpointer.ocn
echo "${refcase}.docn.rs1.${init_time}.bin" >> rpointer.ocn
if ($river_runoff == 'RTM') then
      ln -sf ${stagedir}/${refcase}.rtm${inst_string}.r.${init_time}.nc    ${refcase}.rtm.r.${init_time}.nc
      ln -sf ${stagedir}/${refcase}.rtm${inst_string}.rh0.${init_time}.nc  ${refcase}.rtm.rh0.${init_time}.nc
      echo "${refcase}.rtm.r.${init_time}.nc" >! rpointer.rof
else if ($river_runoff == 'MOSART') then
      ln -sf ${stagedir}/${refcase}.mosart${inst_string}.r.${init_time}.nc   ${refcase}.mosart.r.${init_time}.nc
      ln -sf ${stagedir}/${refcase}.mosart${inst_string}.rh0.${init_time}.nc ${refcase}.mosart.rh0.${init_time}.nc
      echo "${refcase}.mosart.r.${init_time}.nc" >! rpointer.rof
endif
```
See attached for a script which successfully built such a case.
[setup_single_from_ens.txt](https://github.com/ESMCI/cime/files/2474105/setup_single_from_ens.txt)

Also, be careful that the CLM (and other?) packages are identical,
so that the CLM restart files will have identical fields in them.

Resolving patch:
diff --git a/scripts/lib/CIME/Servers/wget.py b/scripts/lib/CIME/Servers/wget.py
--- a/scripts/lib/CIME/Servers/wget.py
+++ b/scripts/lib/CIME/Servers/wget.py
@@ -15,9 +15,9 @@ def __init__(self, address, user='', passwd=''):
             self._args += "--password {} ".format(passwd)
         self._server_loc = address
 
-        err = run_cmd("wget {} --spider {}".format(self._args, address))[0]
-        expect(err == 0,"Could not connect to repo '{0}'\nThis is most likely either a proxy, or network issue .")
-
+        cmd = "wget {} --no-check-certificate --spider {}".format(self._args, address)
+        err, output, _ = run_cmd(cmd, combine_output=True)
+        expect(err == 0,"Could not connect to repo via '{}'\nThis is most likely either a proxy, or network issue.\nOutput:\n{}".format(cmd, output.encode('utf-8')))
 
     def fileexists(self, rel_path):
         full_url = os.path.join(self._server_loc, rel_path)
@@ -30,9 +30,9 @@ def fileexists(self, rel_path):
     def getfile(self, rel_path, full_path):
         full_url = os.path.join(self._server_loc, rel_path)
         stat, output, errput = \
-                run_cmd("wget {} {} -nc --output-document {}".format(self._args, full_url, full_path))
+                run_cmd("wget {} {} -nc --no-check-certificate --output-document {}".format(self._args, full_url, full_path))
         if (stat != 0):
-            logging.warning("wget failed with output: {} and errput {}\n".format(output, errput))
+            logging.warning("wget failed with output: {} and errput {}\n".format(output.encode('utf-8'), errput.encode('utf-8')))
             # wget puts an empty file if it fails.
             try:
                 os.remove(full_path)
@@ -46,7 +46,7 @@ def getfile(self, rel_path, full_path):
     def getdirectory(self, rel_path, full_path):
         full_url = os.path.join(self._server_loc, rel_path)
         stat, output, errput = \
-            run_cmd("wget  {} {} -r -N --no-directories ".format(self._args, full_url+os.sep), from_dir=full_path)
+            run_cmd("wget  {} {} -r -N --no-check-certificate --no-directories ".format(self._args, full_url+os.sep), from_dir=full_path)
         logger.debug(output)
         logger.debug(errput)
         if (stat != 0):


Test patch:
diff --git a/config/e3sm/tests.py b/config/e3sm/tests.py
--- a/config/e3sm/tests.py
+++ b/config/e3sm/tests.py
@@ -48,8 +48,8 @@
 
     "e3sm_atm_integration" : {
         "tests" : (
-            "ERP_Ln9.ne4_ne4.FC5AV1C-L-AQUAP",
-            "SMS_Ld1.ne4_ne4.FC5AV1C-L-AQUAP.cam-clubb_only",
+            "ERP_Ln9.ne4_ne4.F-EAMv1-AQP1",
+            "SMS_Ld1.ne4_ne4.F-EAMv1-AQP1.cam-clubb_only",
             "PET_Ln5.ne4_ne4.FC5AV1C-L.allactive-mach-pet",
             "PEM_Ln5.ne4_ne4.FC5AV1C-L",
             "SMS_D_Ln5.ne4_ne4.FC5AV1C-L.cam-cosplite_nhtfrq5",