Repository: ESMCI/cime

Issue:
Unsupported testlist versions should trigger an error
#2316 (included in [cime5.4.0-alpha.25](https://github.com/ESMCI/cime/releases/tag/cime5.4.0-alpha.25)) dropped support for v1 of the testlist XML file, but trying to run a test suite from a v1.0 testlist file results in no tests being run instead of a traceable error.

For example, POP's test list in both `cesm2_0_alpha10b` and `cesm2_0_alpha10c` is still v1.0, and running `aux_pop` on cheyenne results in:

```
$ qcmd -- ./create_test --parallel-jobs 4 --proc-pool 36 --xml-machine cheyenne --xml-compiler intel --xml-category aux_pop --queue economy --walltime 2:00 --project P93300070 --test-root /glade/scratch/mlevy/tests/empty_testlist
Submitting command to PBS using account P93300070:
    ./create_test --parallel-jobs 4 --proc-pool 36 --xml-machine cheyenne --xml-compiler intel --xml-category aux_pop --queue economy --walltime 2:00 --project P93300070 --test-root /glade/scratch/mlevy/tests/empty_testlist

Waiting for job 7572427.chadmin1 to start ...


Testnames: []
RUNNING TESTS:
Due to presence of batch system, create_test will exit before tests are complete.
To force create_test to wait for full completion, use --wait
At test-scheduler close, state is:
test-scheduler took 0.01305103302 seconds
```

Even though there are 35 tests defined in the no-longer-supported XML format.


Discussion:
Another way to see this is:

```
[roo2:~/cesm_code/cesm/cime/scripts]$ ./query_testlists --xml-category aux_pop
ERROR: No tests found with the following options (where 'None' means no subsetting on that attribute):
	Machine = None
	Category = aux_pop
	Compiler = None
	Testlist = None
```

Personally, I'm less concerned about this doing the right thing for v1 test lists (because I don't expect any components are going to go back to a v1 test list at this point). But I'm more concerned that this may be an indication of a general lack of error checking. For example, if there were some other mistake – including the possibility that there is a typo in the schema version defined at the top of the file – would this silently drop tests?
@billsacks 

> Personally, I'm less concerned about this doing the right thing for v1 test lists (because I don't expect any components are going to go back to a v1 test list at this point)

It's not just about v1, though... what happens when v3 is introduced and then, down the road, v2 is no longer supported?

> I'm more concerned that this may be an indication of a general lack of error checking

This is a good point, although your `query_testlists` example shows that at least one error is reported by some tools but not others.
> It's not just about v1, though... what happens when v3 is introduced and then, down the road, v2 is no longer supported?

Sorry I was unclear: That was the point I was trying to make, too.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/testlist.py b/scripts/lib/CIME/XML/testlist.py
--- a/scripts/lib/CIME/XML/testlist.py
+++ b/scripts/lib/CIME/XML/testlist.py
@@ -52,6 +52,8 @@ def __init__(self,infile, files=None):
             files = Files()
         schema = files.get_schema("TESTS_SPEC_FILE")
         GenericXML.__init__(self, infile, schema=schema)
+        expect(self.get_version() >= 2.0,
+               "{} is an unsupported version of the testfile format and will be ignored".format(infile))
 
     def get_tests(self, machine=None, category=None, compiler=None, compset=None, grid=None, supported_only=False):
         tests = []


Test patch: