Repository: ESMCI/cime

Issue:
Want better organization of xml variables, and more streamlined output from xmlquery
This request emerged from a discussion with @mvertens and @cecilehannay . It is slightly related to #767, in that it touches on the output format from xmlquery.

@cecilehannay and I would like two changes to make it easier to find xml variables of interest and see current values at a glance:

1. A new, streamlined output option from xmlquery that looks like this:

```
category A:
   foo = 1
   bar = 2

category B:
   baz = 3
   etc.
```

i.e., listing category headers and then just giving name/value for each xml variable.

2. Better organization of variables into categories, as well as sorting of categories.

In older versions of CESM, files like env_run.xml were organized so that the most commonly-changed variables were near the top. That is somewhat, but not entirely the case any more. In addition, variables that fit together (such as the various STOP_ variables, or all variables related to a branch run) were visually grouped together in the xml file. Again, this seems to be somewhat the case, but not entirely so.

Here are some examples from cesm1_0_6:

Top of env_run.xml:

```xml
<!--"A setting of TRUE implies a continuation run, valid values: TRUE,FALSE (logical) " -->
<entry id="CONTINUE_RUN"   value="FALSE"  />

<!--"if RESUBMIT is greater than 0, then case will automatically resubmit (integer) " -->
<entry id="RESUBMIT"   value="0"  />

<!-- ====================================== -->

<!--"CCSM tag (char) " -->
<entry id="CCSM_REPOTAG"   value="cesm1_0_6"  />

<!--"case description (char) " -->
<entry id="CASESTR"   value="UNSET"  />

<!-- ====================================== -->

<!--"sets the run length with STOP_N and STOP_DATE, valid values: none,never,nsteps,nstep,nseconds,nsecond,nminutes,nminute,nhours,nhour,ndays,nday,nmonths,nmonth,nyears,nyear,date,ifdays0,end (char) "\
 -->
<entry id="STOP_OPTION"   value="ndays"  />

<!--"sets the run length with STOP_OPTION and STOP_DATE (integer) " -->
<entry id="STOP_N"   value="5"  />

<!--"date in yyyymmdd format, sets the run length with STOP_OPTION and STOP_N (integer) " -->
<entry id="STOP_DATE"   value="-999"  />

<!-- ====================================== -->

<!--"sets frequency of model restart writes (same options as STOP_OPTION) (char) " -->
<entry id="REST_OPTION"   value="$STOP_OPTION"  />

<!--"sets model restart writes with REST_OPTION and REST_DATE (char) " -->
<entry id="REST_N"   value="$STOP_N"  />

<!--"date in yyyymmdd format, sets model restart write date with REST_OPTION and REST_N (char) " -->
<entry id="REST_DATE"   value="$STOP_DATE"  />
```

Top of env_conf.xml:

```xml
<!--"Run initialization type, valid values: startup,hybrid,branch (char) " -->
<entry id="RUN_TYPE"   value="hybrid"  />

<!--"Run start date (yyyy-mm-dd). Only used for startup or hybrid runs (char) " -->
<entry id="RUN_STARTDATE"   value="0001-01-01"  />

<!--"Reference case for hybrid or branch runs (char) " -->
<entry id="RUN_REFCASE"   value="I2000CN_f09_g16_c100503"  />

<!--"Reference date for hybrid or branch runs (yyyy-mm-dd) (char) " -->
<entry id="RUN_REFDATE"   value="0001-01-01"  />

<!--"allow same branch casename as reference casename, valid values: TRUE,FALSE (logical) " -->
<entry id="BRNCH_RETAIN_CASENAME"   value="FALSE"  />

<!--"flag for automatically prestaging the refcase restart dataset, valid values: TRUE,FALSE (logical) " -->
<entry id="GET_REFCASE"   value="TRUE"  />
```

I remember that, in general, I could just look at the top page or so of these files and find everything I needed.



Discussion:
This issue was partly addressed by #930 , but I feel that we should still do more to make this user-friendly.

First, I like the new --listall option, but feel this could use some fine-tuning:

1. It would be easier to read/understand the output if it were separated by file (env_build, env_run, etc.). Currently, results from different files are interleaved (starts with a group from env_run, then gives a group from env_mach_pes.xml, then one from env_build, then another from env_run, etc.). In addition to grouping the output by file, it would also help if it were clear which file each category belonged to. This could be done with a big header line for the start of each file, or by putting the file name in the line giving the group name.

2. Optionally, connected to (1), it could be nice to have an option to just list the variables in a given file, though I'd consider this icing on the cake - not necessary.

3. It would be nice if the "important" groups were listed first. This is done to some extent in the xml files - where (e.g.) run_begin_stop_restart comes near the top of env_run.xml. The output ordering from --listall should ideally match the ordering in the xml files in this respect. (In contrast, the relatively obscure run_pio group was listed first in my output, so my eyes glazed over before I got to the more important stuff.)

4. Similarly to (3), variables should be ordered in a more logical fashion in this output. Again, it would be nice if the ordering here matched the ordering in the xml file. In contrast, my output had the following ordering; notice, for example, the separation of the various `STOP_XXX` variables:

 ```
Results in group run_begin_stop_restart
        BARRIER_DATE: -999
        RUN_REFCASE: case.std
        RUN_WITH_SUBMIT: FALSE
        STOP_OPTION: ndays
        START_TOD: 0
        REST_OPTION: ndays
        RUN_REFTOD: 00000
        BARRIER_OPTION: never
        RESUBMIT: 0
        REST_N: 5
        STOP_DATE: -999
        CONTINUE_RUN: FALSE
        RUN_REFDATE: 0001-01-01
        RUN_TYPE: startup
        RUN_STARTDATE: 0001-01-01
        STOP_N: 5
        BARRIER_N: 1
        RESUBMIT_SETS_CONTINUE_RUN: TRUE
        RUN_REFDIR: ccsm4_init
        REST_DATE: -999
        GET_REFCASE: FALSE
 ```


Finally, on a different note:

5. I don't think #930 did much to address (2) from the original issue: It would still be nice to have more readable xml files - with visual separation between categories - or else more fine-grained categories (along with some extra white space between categories in the case xml files) to achieve the same thing. It's harder to read/skim the case xml files than it was in old versions of the scripts, and I feel it's worth maintaining some better human readability if that can be done without too much trouble.

cc @jedwards4b @mvertens @cecilehannay 
Regarding my comment https://github.com/ESMCI/cime/issues/848#issuecomment-268612559 : For `--listall`: Things are a bit better now in that groups and variables are alphabetized. But this still doesn't go as far in terms of user-friendliness as what I suggested. The most important piece is (1): organizing by file.

And point (5) from that comment hasn't been addressed.
I honestly do not think that item 5 is worthy of pursuing.  Expecting a standard of human readability from these files puts an unnecessary and difficult to maintain restriction on future improvements to the xml format.  
I agree with @jedwards4b on point (5).
XML is human readable and I strongly support ASCII (or at least text based) files for all of our code and configuration source.
However, our 'readability' interface is `xmlquery`, not the XML files. If there are ways to get more user-friendly output from that tool, that is where the effort should go (IMHO).
That said, I don't think we should just shove anything anywhere into an XML file so if something is really out of place, open a specific issue for that.

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -1226,23 +1226,23 @@ def set_file(self, xmlfile):
 
         gfile = GenericXML(infile=xmlfile)
         ftype = gfile.get_id()
-
+        components = self.get_value("COMP_CLASSES")
         logger.warn("setting case file to %s"%xmlfile)
         new_env_file = None
         for env_file in self._env_entryid_files:
             if os.path.basename(env_file.filename) == ftype:
                 if ftype == "env_run.xml":
-                    new_env_file = EnvRun(infile=xmlfile)
+                    new_env_file = EnvRun(infile=xmlfile, components=components)
                 elif ftype == "env_build.xml":
-                    new_env_file = EnvBuild(infile=xmlfile)
+                    new_env_file = EnvBuild(infile=xmlfile, components=components)
                 elif ftype == "env_case.xml":
-                    new_env_file = EnvCase(infile=xmlfile)
+                    new_env_file = EnvCase(infile=xmlfile, components=components)
                 elif ftype == "env_mach_pes.xml":
-                    new_env_file = EnvMachPes(infile=xmlfile)
+                    new_env_file = EnvMachPes(infile=xmlfile, components=components)
                 elif ftype == "env_batch.xml":
-                    new_env_file = EnvBatch(infile=xmlfile)
+                    new_env_file = EnvBatch(infile=xmlfile, components=components)
                 elif ftype == "env_test.xml":
-                    new_env_file = EnvTest(infile=xmlfile)
+                    new_env_file = EnvTest(infile=xmlfile, components=components)
             if new_env_file is not None:
                 self._env_entryid_files = []
                 self._env_generic_files = []


Test patch: