Repository: ESMCI/cime

Issue:
xmlchange fails when setting JOB_QUEUE with --file and --subgroup options
```
cori04% ./xmlchange --file env_batch.xml JOB_QUEUE="regular" --subgroup=case.run
setting case file to env_batch.xml
ERROR: No variable USER_REQUESTED_QUEUE found in case
```
I've tested this on Skybridge, and it only occurs when both --file and --subgroup are specified.


Discussion:
USER_REQUESTED_QUEUE and USER_REQUESTED_WALLTIME are defined in env_run.xml 
and so using the --file env_batch.xml would give the above response as expected.    

Oh - on rereading this - why are you using the --file option?   @jgfouca this would seem to be a problem with locating these variables in env_run.xml but the usage of the --file flag is also unusual.
Should the error message say: "ERROR: No variable USER_REQUESTED_QUEUE found in env_batch.xml"? 
Yes, we can probably improve that error message, however the issue in this case is that the user wasn't directly referring to the USER_REQUESTED variable.
But this also points to another bug:  
```
:) ./xmlchange --subgroup case.st_archive JOB_QUEUE=fred
yslogin3: ~/sandboxes/cesm2_0_alpha/cime/scripts/foo
:) ./xmlquery JOB_QUEUE

Results in group case.run
	JOB_QUEUE: caldera

Results in group case.st_archive
	JOB_QUEUE: fred

Results in group case.test
	JOB_QUEUE: caldera
yslogin3: ~/sandboxes/cesm2_0_alpha/cime/scripts/foo
:) ./case.setup --reset
Successfully cleaned batch script case.run
/glade/p/work/jedwards/sandboxes/cesm2_0_alpha/cime/scripts/foo/env_mach_specific.xml already exists, delete to replace
Creating batch script case.run
Writing case.run script from input template /glade/p/work/jedwards/sandboxes/cesm2_0_alpha/cime/config/cesm/machines/template.case.run
Writing case.st_archive script from input template /glade/p/work/jedwards/sandboxes/cesm2_0_alpha/cime/config/cesm/machines/template.st_archive
WARNING: queue 'fred' is unknown to this system
Locking file env_mach_pes.xml
Creating user_nl_xxx files for components and cpl
If an old case build already exists, might want to run 'case.build --clean' before building
yslogin3: ~/sandboxes/cesm2_0_alpha/cime/scripts/foo
:) ./xmlquery JOB_QUEUE

Results in group case.run
	JOB_QUEUE: fred

Results in group case.st_archive
	JOB_QUEUE: fred

Results in group case.test
	JOB_QUEUE: caldera
```
Suggested solution:  the USER_REQUESTED variables need to be removed from env_run.xml and added to env_batch.xml in each of the job groups.
Is "--file" a legacy argument?  Its never needed, right?

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -183,9 +183,6 @@ def make_batch_script(self, input_template, job, case):
         os.chmod(job, os.stat(job).st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
 
     def set_job_defaults(self, batch_jobs, case):
-        walltime    = case.get_value("USER_REQUESTED_WALLTIME") if case.get_value("USER_REQUESTED_WALLTIME") else None
-        force_queue = case.get_value("USER_REQUESTED_QUEUE") if case.get_value("USER_REQUESTED_QUEUE") else None
-
         if self._batchtype is None:
             self._batchtype = self.get_batch_system_type()
 
@@ -193,6 +190,9 @@ def set_job_defaults(self, batch_jobs, case):
             return
 
         for job, jsect in batch_jobs:
+            walltime    = case.get_value("USER_REQUESTED_WALLTIME", subgroup=job) if case.get_value("USER_REQUESTED_WALLTIME", subgroup=job) else None
+            force_queue = case.get_value("USER_REQUESTED_QUEUE", subgroup=job) if case.get_value("USER_REQUESTED_QUEUE", subgroup=job) else None
+            logger.info("job is {} USER_REQUESTED_WALLTIME {} USER_REQUESTED_QUEUE {}".format(job, walltime, force_queue))
             task_count = jsect["task_count"] if "task_count" in jsect else None
             if task_count is None:
                 node_count = case.num_nodes
@@ -415,6 +415,7 @@ def _submit_single_job(self, case, job, dep_jobs=None, no_batch=False,
             logger.info("dependencies: {}".format(dep_jobs))
             dep_string = self.get_value("depend_string", subgroup=None)
             separator_string = self.get_value("depend_separator", subgroup=None)
+            expect(separator_string is not None,"depend_separator string not defined")
             expect("jobid" in dep_string, "depend_string is missing jobid for prerequisite jobs")
             dep_ids_str = str(dep_jobs[0])
             for dep_id in dep_jobs[1:]:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -391,8 +391,12 @@ def set_value(self, item, value, subgroup=None, ignore_type=False, allow_undefin
                 self._env_files_that_need_rewrite.add(env_file)
                 return result
 
-        expect(allow_undefined or result is not None,
-               "No variable {} found in case".format(item))
+        if len(self._files) == 1:
+            expect(allow_undefined or result is not None,
+                   "No variable {} found in file {}".format(item, self._files[0].filename))
+        else:
+            expect(allow_undefined or result is not None,
+                   "No variable {} found in case".format(item))
 
     def set_valid_values(self, item, valid_values):
         """


Test patch: