Repository: ESMCI/cime

Issue:
FORCE_BUILD_SMP doesn't seem to work right if set after case.setup
At least with CESM, I get build errors in mct if I try to set `./xmlchange FORCE_BUILD_SMP=TRUE` after running case.setup but before building:

```
gmake -f /glade/scratch/sacks/test_threaded_0624b/Tools/Makefile  -C /glade/scratch/sacks/test_threaded_0624b/bld/intel/mpt/nodebug/threads/nuopc/mct CIME_MODEL=cesm  compile_threaded=TRUE CASEROOT="/glade/scratch/sacks/test_threaded_0624b" CASETOOLS="/glade/scratch/sacks/test_threaded_0624b/Tools" CIMEROOT="/glade/work/sacks/ctsm_code/current_branch2/cime" COMP_INTERFACE="nuopc" COMPILER="intel" DEBUG="FALSE" EXEROOT="/glade/scratch/sacks/test_threaded_0624b/bld" INCROOT="/glade/scratch/sacks/test_threaded_0624b/bld/lib/include" LIBROOT="/glade/scratch/sacks/test_threaded_0624b/bld/lib" MACH="cheyenne" MPILIB="mpt" NINST_VALUE="c1a1l1r1" OS="LINUX" PIO_VERSION="2" SHAREDLIBROOT="/glade/scratch/sacks/test_threaded_0624b/bld" SMP_PRESENT="FALSE" USE_ESMF_LIB="TRUE" USE_MOAB="FALSE" COMP_LND="clm" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE" MODEL=mct COMP_NAME=mct /glade/scratch/sacks/test_threaded_0624b/bld/intel/mpt/nodebug/threads/nuopc/mct/Makefile.conf
gmake: Entering directory '/glade/scratch/sacks/test_threaded_0624b/bld/intel/mpt/nodebug/threads/nuopc/mct'
/glade/scratch/sacks/test_threaded_0624b/Tools/mkSrcfiles
cp -f Filepath Deppath
/glade/scratch/sacks/test_threaded_0624b/Tools/mkDepends  Deppath Srcfiles > Depends
gmake: Leaving directory '/glade/scratch/sacks/test_threaded_0624b/bld/intel/mpt/nodebug/threads/nuopc/mct'
cat: Filepath: No such file or directory
cat: Srcfiles: No such file or directory
gmake: *** No rule to make target '/glade/scratch/sacks/test_threaded_0624b/bld/intel/mpt/nodebug/threads/nuopc/mct/Makefile.conf'.  Stop.
ERROR: cat: Filepath: No such file or directory
cat: Srcfiles: No such file or directory
gmake: *** No rule to make target '/glade/scratch/sacks/test_threaded_0624b/bld/intel/mpt/nodebug/threads/nuopc/mct/Makefile.conf'.  Stop.
```

This works if I set `FORCE_BUILD_SMP` before case.setup. It looks like case.setup sets `SMP_PRESENT` based on `FORCE_BUILD_SMP`, but after that point, `FORCE_BUILD_SMP` doesn't seem to be considered. Glancing through cime history, it looks like there were some changes in the last year or two (git blaming / logging indicates 798f179f1, but that doesn't look like the root cause) that may have changed this behavior: There used to be logic like:

```python
    if build_threaded:
        os.environ["SMP"] = "TRUE"
    else:
        os.environ["SMP"] = "FALSE"
```

but that doesn't seem to be present any more.


Discussion:
I should add: The workaround is easy enough (make sure to set `FORCE_BUILD_SMP` before running `./case.setup`), but this still seems like a bug that should be fixed at some point.
This only happens the first time you run case.build, if you run it a second time it works fine - even after a 
case.build --clean-all 
We no longer set SMP in the environment, instead it is passed on the gmake command line.   I don't think that the change you referenced above is relevant.  
> We no longer set SMP in the environment, instead it is passed on the gmake command line. I don't think that the change you referenced above is relevant.

What I meant by bringing this up was that – based on my very quick look – I think the logic may have been correct when this was set in the environment, but some necessary logic may have gotten lost in the big refactor to avoid setting this and other things in the environment.

> This only happens the first time you run case.build, if you run it a second time it works fine - even after a
case.build --clean-all

So would you agree that this is a bug? It seems like needing to run the build twice is not expected behavior. If there is some problem with getting the settings done correctly prior to the first build, then it seems like this variable should be moved to a file that is locked at case.setup time. Do you agree?
I agree that it's a bug, but I'm stumped as to why it is happening.  Your solution seems to me more like a workaround, but I can't think of a case where it would be a problem so I'm okay with that. 


A bit naively, it seems like this logic in `build.py`:

https://github.com/ESMCI/cime/blob/ce4447262300500941a30a64ed73346f37e852f5/scripts/lib/CIME/build.py#L22

is setting the wrong thing. According to the Makefile logic:

https://github.com/ESMCI/cime/blob/ce4447262300500941a30a64ed73346f37e852f5/scripts/Tools/Makefile#L33-L47

the passed-in `compile_threaded` seems to be ignored. Instead, maybe `build.py` should be setting either `SMP` or `SMP_PRESENT`???
But I should add that I really don't understand the difference between all of these `SMP` variables, and how and when they're set, so I should stop making suggestions :-)
@amametjanov might be able to comment here.

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -19,7 +19,7 @@
 
 def get_standard_makefile_args(case, shared_lib=False):
     make_args = "CIME_MODEL={} ".format(case.get_value("MODEL"))
-    make_args += " compile_threaded={} ".format(stringify_bool(case.get_build_threaded()))
+    make_args += " SMP={} ".format(stringify_bool(case.get_build_threaded()))
     if not shared_lib:
         make_args += " USE_KOKKOS={} ".format(stringify_bool(uses_kokkos(case)))
     for var in _CMD_ARGS_FOR_BUILD:


Test patch: