Repository: ESMCI/cime

Issue:
Tweak testlist xml format to facilitate adding test metadata
The current testlist xml format looks like this:

``` xml
  <test name="ERR_N3" grid="f19_g16" compset="B1850" testmods="allactive/defaultio">
    <machines>
      <machine name="edison" compiler="intel" category="prebeta"/>
      <machine name="yellowstone" compiler="intel" category="prealpha"/>
      <options>
    <option name="wallclock"> 00:20 </option>
      </options>
    </machines>
  </test>
```

(First a word of appreciation: This is a lot nicer than the older testlist xml format, which was heavily nested.)

Currently, this supports the option of wallclock. We have also talked about adding options for expected memleak amounts (in #588 ) and expected failures (in some older discussions). But these latter options - and to some extent the wallclock option - are machine and compiler-specific, so it seems like these need to be nested under machine, not in a separate options block that applies to all machines/compilers.

Also, somewhat related: Can someone document how to do the following with the testlist xml files (or point me to this documentation if it's already documented somewhere):
1. Add a comment that appears when using the tool that queries testlists (i.e., not just a comment in the xml, but something with a special comment tag). I don't see any examples of this in the new xml formatted files. Incidentally, this would be another tag that is good to have nested under a specific machine / compiler / category, because my comments are often specific to that level.
2. Have the exact same test appear in multiple categories: Is the correct way to do this to have two <machine...> lines that are identical except for their category, or to have something like <machine name="yellowstone" compiler="intel" category="aux_clm45|prealpha">?
3. Similar question to (2), but how do you have the same test run with multiple compilers? Again, is the expectation that there should be multiple <machine...> lines or one machine line with a '|' in the match?



Discussion:
cc @jedwards4b 

Incidentally: I personally still prefer an even flatter format that more closely matches the test names that are used when creating single tests or examining test results. e.g., something like:

``` xml
<test name="SMS_D_Ly6.1x1_smallvilleIA.IHISTCLM45BGCCROP.yellowstone_pgi.clm-crop_trans_sville">
...
```

Although I can also see some argument for separating out the machine and compiler. This would work best if we made the testmods come before the machine and compiler, so an individual test would look like `SMS_D_Ly6.1x1_smallvilleIA.IHISTCLM45BGCCROP.clm-crop_trans_sville.yellowstone_pgi` and the xml would look like:

``` xml
<test name="SMS_D_Ly6.1x1_smallvilleIA.IHISTCLM45BGCCROP.clm-crop_trans_sville">
  <machine name="yellowstone" compiler="pgi" category="aux_clm45">
...
```

My rationale for these suggestions is motivated by my typical use cases:
1. Adding a new test to a test list
   Here, I usually run the test first as a single test, to make sure it passes, then add it to the list. It's less error-prone if you can simply copy and paste here.
2. Removing a test
   Here, I typically know the exact test (full test name) I want to remove, and then search the plain text file for it. This would be harder to do with the pieces of the test name separated into separate attributes.
3. Modifying one or more existing tests – e.g., changing the run length, grid, etc.
   This use case is similar to (2), in that I typically first do a search for the exact test(s) I want to change. As with (2), this search would be easier if the xml file contained a single string giving the full test name (without machine/compiler).

I realize that munging together all of the aspects of the test name into a single period-delimited string is less clean / elegant than the current format. However, I don't see much practical downside of it myself. It means that create_test needs to parse this string, but that machinery is needed anyway for creation of single tests. It also means that developers need to know how to combine these attributes into a test name, but again: that's needed anyway for the creation of single tests and for understanding test results.

However, all that said: I realize that we had this discussion a year ago and I lost this argument :-)

> Currently, this supports the option of wallclock. We have also talked about adding options for expected memleak amounts (in #588 ) and expected failures (in some older discussions). But these latter options - and to some extent the wallclock option - are machine and compiler-specific, so it seems like these need to be nested under machine, not in a separate options block that applies to all machines/compilers.

Note that the options section can apply to all machines as shown above, or to a particular machine by nesting it one level deeper.

> Note that the options section can apply to all machines as shown above, or to a particular machine by nesting it one level deeper.

Thanks for pointing that out, Jim. 

Okay, assuming that

https://github.com/ESMCI/cime/issues/620#issuecomment-250797536

ain't gonna happen, I think all that's needed here is better documentation. This includes the 3 things I mentioned above, plus now one more... all copied here for one-stop-shopping:

Can someone document how to do the following with the testlist xml files:
1. Add a comment that appears when using the tool that queries testlists (i.e., not just a comment in the xml, but something with a special comment tag). I don't see any examples of this in the new xml formatted files. Incidentally, this would be another tag that is good to have nested under a specific machine / compiler / category, because my comments are often specific to that level.
2. Have the exact same test appear in multiple categories: Is the correct way to do this to have two lines that are identical except for their category, or to have something like ?
3. Similar question to (2), but how do you have the same test run with multiple compilers? Again, is the expectation that there should be multiple lines or one machine line with a '|' in the match?
4. Document the different ways you can specify options - either nested under a machine tag or not.

I'm not sure what the best place would be for this documentation, but testlist.py seems like a reasonable option.

Oops, my offline writing crossed paths with Jim's PR #626 . To keep things clear, I'll change this back to the old title, and open a new issue regarding documentation.


Resolving patch:
diff --git a/utils/python/CIME/XML/testlist.py b/utils/python/CIME/XML/testlist.py
--- a/utils/python/CIME/XML/testlist.py
+++ b/utils/python/CIME/XML/testlist.py
@@ -1,6 +1,12 @@
 """
 Interface to the config_files.xml file.  This class inherits from generic_xml.py
 It supports versions 1.0 and 2.0 of the testlist.xml file
+
+In version 2 of the file options can be specified to further refine a test or set of tests
+they can be specified at the <machines> level or at the level of a particular machine.
+currently supported options are walltime which sets the wallclock time for a given test
+in the queing system, and memleak_tolerance which specifies the relative memory growth expected for a given test.
+
 """
 from CIME.XML.standard_module_setup import *
 
@@ -79,10 +85,17 @@ def _get_testsv2(self, machine=None, category=None, compiler=None):
                             this_test["machine"] = value
                         else:
                             this_test[key] = value
+                    this_test["options"] = {}
+                    # option xpath here gets only the children of tnode and ignores grandchildren
+                    optionnodes = self.get_nodes("option", root=tnode, xpath="options/option")
 
+                    for onode in optionnodes:
+                        this_test['options'][onode.get('name')] = onode.text
                     optionnodes = self.get_nodes("option", root=mach)
                     for onode in optionnodes:
-                        this_test[onode.get('name')] = onode.text
+                        this_test['options'][onode.get('name')] = onode.text
+
+
                     tests.append(this_test)
 
         return tests


Test patch:
diff --git a/utils/python/CIME/SystemTests/system_tests_common.py b/utils/python/CIME/SystemTests/system_tests_common.py
--- a/utils/python/CIME/SystemTests/system_tests_common.py
+++ b/utils/python/CIME/SystemTests/system_tests_common.py
@@ -288,10 +288,13 @@ def _check_for_memleak(self):
                 memdiff = -1
                 if originalmem > 0:
                     memdiff = (finalmem - originalmem)/originalmem
-
+                tolerance = self._case.get_value("TEST_MEMLEAK_TOLERANCE")
+                if tolerance is None:
+                    tolerance = 0.1
+                expect(tolerance > 0.0, "Bad value for memleak tolerance in test")
                 if memdiff < 0:
                     self._test_status.set_status(MEMLEAK_PHASE, TEST_PASS_STATUS, comments="insuffiencient data for memleak test")
-                elif memdiff < 0.1:
+                elif memdiff < tolerance:
                     self._test_status.set_status(MEMLEAK_PHASE, TEST_PASS_STATUS)
                 else:
                     comment = "memleak detected, memory went from %f to %f in %d days" % (originalmem, finalmem, finaldate-originaldate)
diff --git a/utils/python/CIME/test_scheduler.py b/utils/python/CIME/test_scheduler.py
--- a/utils/python/CIME/test_scheduler.py
+++ b/utils/python/CIME/test_scheduler.py
@@ -347,8 +347,9 @@ def _create_newcase_phase(self, test):
 
         if self._walltime is not None:
             create_newcase_cmd += " --walltime %s" % self._walltime
-        elif test in self._test_data and "wallclock" in self._test_data[test]:
-            create_newcase_cmd += " --walltime %s" % self._test_data[test]['wallclock']
+        elif test in self._test_data and "options" in self._test_data[test] and \
+                "wallclock" in self._test_data[test]['options']:
+            create_newcase_cmd += " --walltime %s" % self._test_data[test]['options']['wallclock']
 
         logger.debug("Calling create_newcase: " + create_newcase_cmd)
         return self._shell_cmd_for_phase(test, create_newcase_cmd, CREATE_NEWCASE_PHASE)
@@ -372,6 +373,9 @@ def _xml_phase(self, test):
         envtest.set_value("TESTCASE", test_case)
         envtest.set_value("TEST_TESTID", self._test_id)
         envtest.set_value("CASEBASEID", test)
+        if test in self._test_data and "options" in self._test_data[test] and \
+                "memleak_tolerance" in self._test_data[test]['options']:
+            envtest.set_value("TEST_MEMLEAK_TOLERANCE", self._test_data[test]['options']['memleak_tolerance'])
 
         test_argv = "-testname %s -testroot %s" % (test, self._test_root)
         if self._baseline_gen_name: