Repository: ESMCI/cime

Issue:
Request for test names with mods to not require machine/compiler strings 
I posted as E3SM issue, but Jim F suggested I post here.

For example, these two tests:

```
ERS.f19_g16.I1850CNECACNTBC.cori-knl_gnu.clm-eca
SMS_D_Ln5.ne4_ne4.FC5AV1C-L.cori-knl_intel18.cam-cosplite_nhtfrq5

would be better if it were:

ERS.f19_g16.I1850CNECACNTBC.clm-eca
SMS_D_Ln5.ne4_ne4.FC5AV1C-L.cam-cosplite_nhtfrq5

cori07% ./create_test ERS.f19_g16.I1850CNECACNTBC.clm-eca --machine=cori-knl
ERROR: Expected 4th item of 'ERS.f19_g16.I1850CNECACNTBC.clm-eca' ('clm-eca') to be in form machine_compiler
```

If a test is not using any xml mods (i.e. ERS.f19_g16.I1850CNECACNTBC.cori-knl_gnu) then one can simply run a test by ./create_test ERS.f19_g16.I1850CNECACNTBC (i.e. no need to specify machine + compiler option). But, if a test is using test mod (ERS.f19_g16.I1850CNECACNTBC.cori-knl_gnu.clm-eca), then one needs to specify the full test name 

`./create_test ERS.f19_g16.I1850CNECACNTBC.cori-knl_gnu.clm-eca`


Discussion:
@jedwards4b , how hard would be it to change test names to be:

TYPE.GRID.COMPET.[TESTMOD].machine_compiler

?
It would require a lot of changes to the cesm testlist xml files unless it could somehow be made backward compatible.   @billsacks had some ideas regarding testlists, Bill? 
@jedwards4b I did have some ideas regarding testlists, but they were pretty orthogonal to this discussion: My ideas revolve around having the full test name listed as a single string, using whatever format of testname string we use.

@jgfouca I don't understand your proposal: if someone specifies a test like `SMS.f09_g16.B1850.foo_bar`, how does the test system know if that's a test with the testmod directory `foo_bar` vs. a test with no testmod running on machine `foo`, compiler `bar`? I suppose it could try one and, if that doesn't work, then try the other. But that hardly leaves me feeling warm and fuzzy.

I thought @gold2718 had a good suggestion when he and I discussed this recently: He suggested having a string format that allows order to not matter – e.g., something like `t%SMS_D.g%f19_g17.c%B1850.m%cheyenne_intel.d%clm-eca` – which would be the same as `m%cheyenne_intel.g%f19_g17.t%SMS_D.c%B1850.d%clm-eca` (and many other orderings). (I wouldn't necessarily expect the test system to be smart enough to find the right baseline with reordering... though maybe it would, by normalizing the string into a particular order.) It seems like this would robustly solve the problem of optional components.

I agree with @jedwards4b that the hardest problem here is probably going to be backwards compatibility, which I think will at least be important for some period of time.
The problem is that we have two optional arguments machine_compiler and modifier and no way to tell the difference between them.   What if we required a double dot to indicate default machine?
That is ./create_test ERS.f19_g16.I1850CNECACNTBC..clm-eca?
But I already see that @ndkeen would not save anything in this case because cori-knl is not the default cori machine and gnu is not the default compiler so in this case his proposed change would save him nothing.
I like the idea of double dot. Or being more explicit, like using a machine_compiler of `default`.
I prefer @billsacks suggestion of using a word like "default" to describe that it's a default machine/compiler.
@billsacks , I think we could make CIME smart enough to recognize the difference. For example, this algorithm:

```
TEST.GRID.COMPSET.UNKNOWN
if UNKNOWN is in format X_Y and X is a known CIME machine and Y is a known compiler:
   treat unknown as MACHINE_COMPILER
else:
   treat unknown as a testmod
```
The only problem would be if someone made a testmod with a name MACHINE_COMPILER.
@jgfouca yeah, I see your point. However, I like the other suggestions on the table better, because they are more extensible to other optional components that may be added to the test name in the future.
@billsacks good point. I wasn't thinking of extensibility.
Do we have an agreed-upon solution?
I'll second my own suggestion of using a special keyword for `machine_compiler` to support this use case. Based on @ekluzek 's points in #2197 , I wonder if something like `all` or `any` would make more sense than `default`. We might have to give some thought to the exact semantics, but I'm thinking something like:

1. If you're running this as a single test (`create_test TESTNAME`), it will run on the current machine with the default compiler

2. If this is defined in a text-based test list (`create_test` with the `-f` option), it will run on the current machine with all supported compilers (e.g., if there are three supported compilers on this machine, three tests will be run)

3. For CESM: If this is defined in an xml (or later, cfg)-based test list: If you're running without a `--compiler`/`--xml-compiler` option, it runs on the current machine on all supported compilers. If you're running with a `--compiler`/`--xml-compiler` option, it runs on the current machine with the specified compiler.

I realize this may be getting more complex than the original issue warrants, so we could move some of this functionality to a separate issue.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.
This issue was closed because it has been stalled for 5 days with no activity.
Reopening because I think resolving this issue would add value. That said, changing the test case name format scares me because there are probably assumptions all over the code about its current form.
What if we added something like ```SMS.f19_g17.A.*_*.cam-outfrq9s```   where * in this case means default?

Resolving patch:
diff --git a/CIME/utils.py b/CIME/utils.py
--- a/CIME/utils.py
+++ b/CIME/utils.py
@@ -271,7 +271,7 @@ def _read_cime_config_file():
     cime_config_file = os.path.abspath(
         os.path.join(os.path.expanduser("~"), ".cime", "config")
     )
-    cime_config = configparser.SafeConfigParser()
+    cime_config = configparser.ConfigParser()
     if os.path.isfile(cime_config_file):
         cime_config.read(cime_config_file)
         for section in cime_config.sections():
@@ -661,8 +661,8 @@ def import_and_run_sub_or_cmd(
             run_sub_or_cmd(
                 cmd, cmdargs, subname, subargs, logfile, case, from_dir, timeout
             )
-        except Exception as e:
-            raise e from None
+        except Exception as e1:
+            raise e1 from None
     except Exception:
         if logfile:
             with open(logfile, "a") as log_fd:
@@ -1014,6 +1014,12 @@ def parse_test_name(test_name):
     ['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', None]
     >>> parse_test_name('ERS.fe12_123.JGF.machine_compiler.test-mods')
     ['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', ['test/mods']]
+    >>> parse_test_name('ERS.fe12_123.JGF.*_compiler.test-mods')
+    ['ERS', None, 'fe12_123', 'JGF', None, 'compiler', ['test/mods']]
+    >>> parse_test_name('ERS.fe12_123.JGF.machine_*.test-mods')
+    ['ERS', None, 'fe12_123', 'JGF', 'machine', None, ['test/mods']]
+    >>> parse_test_name('ERS.fe12_123.JGF.*_*.test-mods')
+    ['ERS', None, 'fe12_123', 'JGF', None, None, ['test/mods']]
     >>> parse_test_name('ERS.fe12_123.JGF.machine_compiler.test-mods--other-dir-path--and-one-more')
     ['ERS', None, 'fe12_123', 'JGF', 'machine', 'compiler', ['test/mods', 'other/dir/path', 'and/one/more']]
     >>> parse_test_name('SMS.f19_g16.2000_DATM%QI.A_XLND_SICE_SOCN_XROF_XGLC_SWAV.mach-ine_compiler.test-mods') # doctest: +IGNORE_EXCEPTION_DETAIL
@@ -1047,6 +1053,10 @@ def parse_test_name(test_name):
             ),
         )
         rv[4:5] = rv[4].split("_")
+        if rv[4] == "*":
+            rv[4] = None
+        if rv[5] == "*":
+            rv[5] = None
         rv.pop()
 
     if rv[-1] is not None:
@@ -1139,7 +1149,6 @@ def get_full_test_name(
     ]
 
     result = partial_test
-
     for partial_val, arg_val, name in required_fields:
         if partial_val is None:
             # Add to result based on args
@@ -1149,9 +1158,14 @@ def get_full_test_name(
                     partial_test, name
                 ),
             )
-            result = "{}{}{}".format(
-                result, "_" if name == "compiler" else ".", arg_val
-            )
+            if name == "machine" and "*_" in result:
+                result = result.replace("*_", arg_val + "_")
+            elif name == "compiler" and "_*" in result:
+                result = result.replace("_*", "_" + arg_val)
+            else:
+                result = "{}{}{}".format(
+                    result, "_" if name == "compiler" else ".", arg_val
+                )
         elif arg_val is not None and partial_val != partial_compiler:
             expect(
                 arg_val == partial_val,


Test patch: