Repository: ESMCI/cime

Issue:
Attempt to download inputdata files with wget is hanging
The check_input_data script is hanging when attempting to check the server with protocol wget.

This problem was originally reported in the CESM forums: https://xenforo.cgd.ucar.edu/cesm/threads/downloading-input-data-b1850-compset-no-server-response.4999/ . I have reproduced it on cheyenne as follows:

```
git clone https://github.com/ESCOMP/CESM.git
cd CESM
git checkout cesm2.1-alphabranch
./manage_externals/checkout_externals
cd cime/scripts/
./create_newcase --case /glade/scratch/sacks/test_inputdata_0124a --compset B1850 --res f09_g17
cd /glade/scratch/sacks/test_inputdata_0124a
mkdir /glade/scratch/sacks/cesm_test_inputdata
./xmlchange DIN_LOC_ROOT=/glade/scratch/sacks/cesm_test_inputdata
./case.setup
./preview_namelists
```

The last command produces:

```
Setting resource.RLIMIT_STACK to -1 from (307200000, -1)
Refcase not found in /glade/scratch/sacks/cesm_test_inputdata/cesm2_init/b.e20.B1850.f09_g17.pi_control.all.299_merge_v3/0134-01-01, will attempt to download from inputdata
Checking server ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/ with protocol gftp
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Client protocol gftp not enabled
Checking server ftp://ftp.cgd.ucar.edu/cesm/inputdata with protocol wget
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol wget with user anonymous and passwd user@example.edu
```

then hangs.

It's possible that the solution to this problem is simply to reboot the system used with wget; that would address the "CRITICAL" nature of this issue.

However, I wonder if it would be possible for the check_input_data script to handle timeouts more gracefully. Now that we support more inputdata servers, it seems like it will be more common for one of them to be down. If the data can be retrieved from the servers that are up, then it seems like the scripts should gracefully handle an unnecessary server being down, and just move on without complaint. (But maybe I'm misunderstanding the problem here, or what I'm suggesting is harder than I think.)


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -224,7 +224,8 @@ def set_job_defaults(self, batch_jobs, case):
 
             walltime    = case.get_value("USER_REQUESTED_WALLTIME", subgroup=job) if case.get_value("USER_REQUESTED_WALLTIME", subgroup=job) else None
             force_queue = case.get_value("USER_REQUESTED_QUEUE", subgroup=job) if case.get_value("USER_REQUESTED_QUEUE", subgroup=job) else None
-            logger.info("job is {} USER_REQUESTED_WALLTIME {} USER_REQUESTED_QUEUE {}".format(job, walltime, force_queue))
+            walltime_format = case.get_value("walltime_format", subgroup=job) if case.get_value("walltime_format", subgroup=job) else None
+            logger.info("job is {} USER_REQUESTED_WALLTIME {} USER_REQUESTED_QUEUE {} WALLTIME_FORMAT {}".format(job, walltime, force_queue, walltime_format))
             task_count = int(jsect["task_count"]) if "task_count" in jsect else case.total_tasks
             walltime = jsect["walltime"] if ("walltime" in jsect and walltime is None) else walltime
             if "task_count" in jsect:
@@ -264,6 +265,18 @@ def set_job_defaults(self, batch_jobs, case):
 
                 walltime = self._default_walltime if walltime is None else walltime # last-chance fallback
 
+            else:
+                # Set the walltime to the correct walltime_format, if set.
+                # Assuming correct format is #H:#M or %H:%M:%S.
+                if walltime_format is not None:
+                    components=walltime.split(":")
+                    if len(components) > len(walltime_format.split(":")):
+                        walltime = ':'.join(components[:len(walltime_format.split(":"))])
+                        logger.info(" Changing USER_REQUESTED_WALLTIME to {} to match walltime_format {}".format(walltime,walltime_format))
+                    if len(components) < len(walltime_format.split(":")):
+                        walltime = walltime + ':' + ':'.join(["00"]*(len(walltime_format.split(":")) - len(components)))
+                        logger.info(" Changing USER_REQUESTED_WALLTIME to {} to match walltime_format {}".format(walltime,walltime_format))
+
             env_workflow.set_value("JOB_QUEUE", self.text(queue), subgroup=job, ignore_type=specs is None)
             env_workflow.set_value("JOB_WALLCLOCK_TIME", walltime, subgroup=job)
             logger.debug("Job {} queue {} walltime {}".format(job, self.text(queue), walltime))
diff --git a/scripts/lib/CIME/nmlgen.py b/scripts/lib/CIME/nmlgen.py
--- a/scripts/lib/CIME/nmlgen.py
+++ b/scripts/lib/CIME/nmlgen.py
@@ -494,7 +494,7 @@ def create_stream_file_and_update_shr_strdata_nml(self, config, caseroot, #pylin
                     continue
                 filepath, filename = os.path.split(filename)
                 if not filepath:
-                    filepath = os.path.join(domain_filepath, os.path.dirname(filename.strip()))
+                    filepath = os.path.join(domain_filepath, filename.strip())
                 string = "domain{:d} = {}\n".format(i+1, filepath)
                 hashValue = hashlib.md5(string.rstrip().encode('utf-8')).hexdigest()
                 if hashValue not in lines_hash:
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -866,7 +866,7 @@ def safe_copy(src_path, tgt_path, preserve_meta=True):
         if not os.access(tgt_path, os.W_OK):
             if owner_uid == os.getuid():
                 # I am the owner, make writeable
-                os.chmod(st.st_mode | statlib.S_IWRITE)
+                os.chmod(tgt_path, st.st_mode | statlib.S_IWRITE)
             else:
                 # I won't be able to copy this file
                 raise OSError("Cannot copy over file {}, it is readonly and you are not the owner".format(tgt_path))


Test patch: