Repository: ESMCI/cime

Issue:
python logger debug statements aren't printed when used in "buildlib".
While updating CAM's buildlib script, it was found that `logger.debug` statements are not printed to any file in the case, even if the log level is set to debug in buildlib itself.  However, `logger.info` statements do make it to the `*.bldlog.*` file. 

This seems to hold even if the `DEBUG` variable is set to True in `env_build.xml`, and if the `--debug` flag is provided to `case.build`.  It also doesn't seem to be CAM-specific, as the same behavior happens in CLM's buildlib script as well (although I didn't test other components' buildlib scripts).

I specifically used the CESM `release-cesm2.1.3` tag on Izumi to run these tests (although I ran a similar test on Cheyenne with the same results).

Any ideas as to why this is the happening would be greatly appreciated.  Thanks!


Discussion:
The buildlib python script in your cime_config directory should have a subroutine called from main.

def buildlib(case, libroot, bldroot, compname=None):

Doing this should cause this to be called as a subroutine instead of a subprocess and solve the issue.    


@jedwards4b I know that you and I came up with the above on our call, but I dug a little more afterwards and I actually don't think that's right. It looks to me like there are separate functions in cime for calling the buildlib for cime's lower-level libraries vs. calling the buildlib for components.

For lower-level libraries (csm_share, etc.), it uses a function call if possible, with arguments `[full_lib_path, os.path.join(exeroot, sharedpath), case]` (from build.py: `_build_libraries`). However, for component builds, it looks like it's using the function `_build_model_thread`, and for that I don't see any logic for calling a component's buildlib as a function. But I may be missing something.
@nusbaume the broader issue seems to be the following (deduced during today's cime call): if a component's buildlib is called as a subprocess, then the logging options don't get passed down. The ideal solution is for a component's buildlib to be called as a python function rather than a subprocess. Our first look at this suggested that this should be possible if you have a correctly-named function, but now I'm thinking that cime's build function needs a little more logic added to do this properly.
@billsacks thanks - I'll take a closer look.
@billsacks @jedwards4b Thanks for the info!  I tried the first suggestion by adding this code to CAM's buildlib main call:

```
if __name__ == "__main__":

    caseroot, libroot, bldroot = parse_input(sys.argv)
    with Case(caseroot) as case:
        buildlib(case, caseroot, libroot, bldroot, "cam")
```
where the function `buildlib` replaced the original `_build_cam`.  Sadly it didn't seem to fix the issue, which could mean either there needs to be additional logic, or I didn't apply your fix correctly (which seems just as likely).


@nusbaume there is something inside cime that needs to be fixed before this will work correctly, but the code change above is heading in the correct direction, it should look like the following,
you want the 'with Case' statement inside the buildlib subroutine. 
```
def buildlib(caseroot, libroot, bldroot):
    with Case(caseroot) as case:
       ...

if __name__ == "__main__":
    caseroot, libroot, bldroot = parse_input(sys.argv)
    buildlib(caseroot, libroot, bldroot)
```
@jedwards4b thanks for the pointer!  I'll wait until I hear about the cime fix, and then try again with the updated `main` statement.
@jgfouca I remember the issue here - the buildlib scripts for external components are called from subthreads and there are some variables communicated to the underlying Makefile through the environment,  environment variables are not thread safe and so we broke the feature of calling as a subroutine in order to solve the issue of setting environment variables independently in each thread.   The variables in question are SMP, MODEL, COMP_CLASS and COMP_NAME, how can we rework this so that both features work?   Or alternatively how do we forward the --debug option if used to the buildlib scripts?
I tried passing in the --debug flag, verified that it did in fact change the logger level setting and I still don't get output from logger.debug("HERE I AM")
@jedwards4b Hi Jim, I got your email and I will take a closer look.

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -3,7 +3,7 @@
 """
 import glob, shutil, time, threading, subprocess, imp
 from CIME.XML.standard_module_setup  import *
-from CIME.utils                 import get_model, analyze_build_log, stringify_bool, run_and_log_case_status, get_timestamp, run_sub_or_cmd, run_cmd, get_batch_script_for_job, gzip_existing_file, safe_copy
+from CIME.utils                 import get_model, analyze_build_log, stringify_bool, run_and_log_case_status, get_timestamp, run_sub_or_cmd, run_cmd, get_batch_script_for_job, gzip_existing_file, safe_copy, check_for_python, get_logging_options
 from CIME.provenance            import save_build_provenance as save_build_provenance_sub
 from CIME.locked_files          import lock_file, unlock_file
 
@@ -423,6 +423,11 @@ def _build_model_thread(config_dir, compclass, compname, caseroot, libroot, bldr
     if get_model() != "ufs":
         compile_cmd = "SMP={} {}".format(stringify_bool(smp), compile_cmd)
 
+    if check_for_python(cmd):
+        logging_options = get_logging_options()
+        if logging_options != "":
+            compile_cmd = compile_cmd + logging_options
+
     with open(file_build, "w") as fd:
         stat = run_cmd(compile_cmd,
                        from_dir=bldroot,  arg_stdout=fd,
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -357,6 +357,19 @@ def _convert_to_fd(filearg, from_dir, mode="a"):
 
 _hack=object()
 
+def check_for_python(filepath, funcname=None):
+    is_python = is_python_executable(filepath)
+    has_function = True
+    if funcname is not None:
+        has_function = False
+        with open(filepath, 'r') as fd:
+            for line in fd.readlines():
+                if re.search(r"^def\s+{}\(".format(funcname), line) or re.search(r"^from.+import.+\s{}".format(funcname), line):
+                    has_function = True
+                    break
+
+    return is_python and has_function
+
 def run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile=None, case=None,
                    from_dir=None, timeout=None):
     """


Test patch: