Repository: ESMCI/cime

Issue:
Naming conventions for files in cime_config/
## Background

In recent CSEG meetings, we talked about file naming conventions to maintain consistency among all the components -- namely, should python files have the `.py` extension or not. The prevailing view was that python modules that are meant to be imported should have the extension, scripts that should be run from the command line should not have it, and files that currently do double duty via `if __name__ == "__main__"` blocks should be split into two files (a `.py` module and a extension-free wrapper).

One big hitch to this plan is how CIME treats `cime_config/buildnml`. Currently there is logic that tries to import specific functions from `buildnml`, and falls back to executing the script via `subprocess`. @billsacks and I chatted about how to modify CIME to follow these conventions, and came up with some interesting ideas we thought were worth discussing.

## Potential paths forward

Assuming other groups agree that this would be a nice convention to follow, there a few possible ways to proceed.

1. The simplest fix would be to update `run_sub_or_cmd()` to try to import `f"{filename}.py"` before falling back on the current behavior of trying to import `filename` and running `filename` if the import fails. Perhaps in a later version this could collapse to "try to import `filename.py` or run `filename`", but leaving the middle check would maintain backwards compatibility for the current version.
2. Another possibility would be to allow components to make `cime_config` a package (i.e. include `__init__.py`). Then we could try to import `cime_config`, and fall back to the current behavior in CIME6 (again, perhaps cutting out the `import filename` option in future versions). This could potentially clean up some python code in components -- one example is that POP has a python layer that sits between `buildnml`; if these functions are included in the `cime_config` package it is easier to import them into `buildnml`.
3. A third possibility would be to enforce a rule that each component has `cime_config/${COMPNAME}/__init__.py`; this would let us more cleanly separate the python code in `cime_config` from the XML and the test suite files. It might also let us clean up `import` statements since there is no longer a namespace conflict every time you import a new `buildnml` (or `cime_config` in the above option)

## Questions

1. Does it make sense to make `buildnml` a wrapper script that imports `buildnml.py`?
2. If so, are any of the above paths the right answer for how CIME should make use of this renamed module?

In case it changes how you answer the above questions, I'll note that I am happy to take lead on implementing whatever solution results from this issue ticket.


Discussion:
@mnlevy1981 thanks for taking this on - I'm interested in seeing what a prototype of each of the options would look like.  Maybe start with option 3 (or another if others feel strongly about it).
I would also lean slightly towards (3) (then (2) then (1)). But in thinking about this a little more, I'd prefer if the package name were more specific than `${COMPNAME}`. This is mostly for selfish ctsm-centric reasons, because we already have a python package named `ctsm` (in https://github.com/ESCOMP/CTSM/tree/master/python), but I could imagine the same being true for other components. I think that, even if we had a true package under `cime_config`, we'd want to keep our core ctsm python in `python/ctsm` because it's a more intuitive place for people to find it. So then we could have import conflicts if we have two packages named `ctsm` in different places.

So maybe option (3) but a path like `cime_config/${COMPNAME}_cime_config_py/__init__.py`? (I'm not sure about the final `_py` after `${COMPNAME}_cime_config`: I added it to make it clear to people browsing the code that this is the directory where you can find python modules, but I don't feel strongly about it.)
Other suggestions from telecon:
${COMPNAME}_cime_py
${COMPNAME}_cimepck_py

NOT ${COMPNAME}_py because another model may use that in some other context.

We all liked option 3 and just need to settle on a name.
I think that the ideas proposed here will also make for a cleaner method of importing system test modules that are defined in components. For example, see this comment that I wrote a while ago:

https://github.com/ESCOMP/CTSM/blob/09ef76bddcd4681bff4cec6b1043b1f8171f8722/cime_config/SystemTests/lii2finidatareas.py#L44-L57

Resolving patch:
diff --git a/scripts/lib/CIME/case/preview_namelists.py b/scripts/lib/CIME/case/preview_namelists.py
--- a/scripts/lib/CIME/case/preview_namelists.py
+++ b/scripts/lib/CIME/case/preview_namelists.py
@@ -4,7 +4,7 @@
 """
 
 from CIME.XML.standard_module_setup import *
-from CIME.utils import run_sub_or_cmd, safe_copy
+from CIME.utils import import_and_run_sub_or_cmd, safe_copy
 import time, glob
 logger = logging.getLogger(__name__)
 
@@ -77,17 +77,10 @@ def create_namelists(self, component=None):
         else:
             compname = self.get_value("COMP_{}".format(model_str.upper()))
         if component is None or component == model_str or compname=="ufsatm":
-            # first look in the case SourceMods directory
-            cmd = os.path.join(caseroot, "SourceMods", "src."+compname, "buildnml")
-            if os.path.isfile(cmd):
-                logger.warning("\nWARNING: Using local buildnml file {}\n".format(cmd))
-            else:
-                # otherwise look in the component config_dir
-                cmd = os.path.join(config_dir, "buildnml")
-            expect(os.path.isfile(cmd), "Could not find buildnml file for component {}".format(compname))
+            cmd = os.path.join(config_dir, "buildnml")
             logger.info("Create namelist for component {}".format(compname))
-            run_sub_or_cmd(cmd, (caseroot), "buildnml",
-                           (self, caseroot, compname), case=self)
+            import_and_run_sub_or_cmd(cmd, (caseroot), "buildnml",
+                           (self, caseroot, compname), config_dir, compname, case=self)
 
         logger.debug("Finished creating component namelists, component {} models = {}".format(component, models))
 
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -422,6 +422,28 @@ def file_contains_python_function(filepath, funcname):
 
     return has_function
 
+def import_and_run_sub_or_cmd(cmd, cmdargs, subname, subargs, config_dir, compname,
+                              logfile=None, case=None, from_dir=None, timeout=None):
+    sys_path_old = sys.path
+    sys.path.insert(1, config_dir)
+    try:
+        mod = importlib.import_module(f"{compname}_cime_py")
+        getattr(mod, subname)(*subargs)
+    except (ModuleNotFoundError, AttributeError) as _:
+        # * ModuleNotFoundError if importlib can not find module,
+        # * AttributeError if importlib finds the module but
+        #   {subname} is not defined in the module
+        expect(os.path.isfile(cmd), f"Could not find {subname} file for component {compname}")
+        run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile, case, from_dir, timeout)
+    except Exception:
+        if logfile:
+            with open(logfile, "a") as log_fd:
+                log_fd.write(str(sys.exc_info()[1]))
+            expect(False, "{} FAILED, cat {}".format(cmd, logfile))
+        else:
+            raise
+    sys.path = sys_path_old
+
 def run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile=None, case=None,
                    from_dir=None, timeout=None):
     """


Test patch: