Repository: ESMCI/cime

Issue:
Don't count all tests when determining if a compset is unsupported
In the determination of whether a compset is listed as unsupported, currently all tests are treated as equal, and if any test exists for it, it is no longer "unsupported".  This causes issues with tests that are set up for development purposes, but are not ready for public use.  I would propose that the counting of tests only count tests from a subset of all the possible test names.  

For example, in CAM's case, we run "aux_cam" before we make a CAM tag.   Then "prealpha" and "prebeta" are also run prior to various CESM tags.  Any other compsets are not routinely tested and could cause issues for an end-user and would thus be identified as "unsupported".  

The list of supported tests (in CAM's case: aux_cam, prealpha, prebeta)  could be maintained in each component's cime_config directory to leave its definition under our control.


Discussion:

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -1113,8 +1113,13 @@ def _check_testlists(self, compset_alias, grid_name, files):
 
         tests = Testlist(tests_spec_file, files)
         testlist = tests.get_tests(compset=compset_alias, grid=grid_name)
-        if len(testlist) > 0:
-            logger.info("\nThis compset and grid combination is not scientifically supported, however it is used in %d tests.\n"%(len(testlist)))
+        testcnt = 0
+        for test in testlist:
+            if test["category"] == "prealpha" or test["category"] == "prebeta" or "aux_" in test["category"]:
+                testcnt += 1
+        if testcnt > 0:
+            logger.info("\nThis compset and grid combination is not scientifically supported, however it is used in %d tests.\n"%(testcnt))
         else:
-            expect(False, "\nThis compset and grid combination is unsupported in CESM.  "
-                   "If you wish to use it anyway you must supply the --run-unsupported option to create_newcase.")
+            expect(False, "\nThis compset and grid combination is untested in CESM.  "
+                   "Override this warning with the --run-unsupported option to create_newcase.",
+                   error_prefix="STOP: ")
diff --git a/utils/python/CIME/utils.py b/utils/python/CIME/utils.py
--- a/utils/python/CIME/utils.py
+++ b/utils/python/CIME/utils.py
@@ -8,7 +8,7 @@
 TESTS_FAILED_ERR_CODE = 100
 logger = logging.getLogger(__name__)
 
-def expect(condition, error_msg, exc_type=SystemExit):
+def expect(condition, error_msg, exc_type=SystemExit, error_prefix="ERROR: "):
     """
     Similar to assert except doesn't generate an ugly stacktrace. Useful for
     checking user error, not programming error.
@@ -23,7 +23,7 @@ def expect(condition, error_msg, exc_type=SystemExit):
         if logger.isEnabledFor(logging.DEBUG):
             import pdb
             pdb.set_trace()
-        raise exc_type("ERROR: %s" % error_msg)
+        raise exc_type("%s %s" % (error_prefix,error_msg))
 
 def id_generator(size=6, chars=string.ascii_lowercase + string.digits):
     return ''.join(random.choice(chars) for _ in range(size))


Test patch: