Repository: ESMCI/cime

Issue:
Wanted: command to clean the Srcfiles and Depends files for a component
At least in some circumstances, if you rebuild a case (i.e., it already has been built, then you rerun `./case.build`) after adding a new source file, the build fails because the new source file and the changed dependency tree are not picked up.

For example, I added a file named NumericsMod.F90 to the CLM source tree, and added `use NumericsMod` in SoilHydrologyMod.F90. Then I reran `./case.build` and got:

```
   use NumericsMod       , only : truncate_small_values
      1
Fatal Error: Can't open module file 'numericsmod.mod' for reading at (1): No such file or directory
compilation terminated.
```

We discussed this on a recent cime concall, and felt that it would be too hard to automatically detect this situation. What I'm asking for here is something simpler: It would be great if there were some `./case.build` option that reset a given component's build _just enough_ to allow picking up any new source files and regenerating the dependency tree. From a quick look at the CESM Makefile, I think this would involve a new set of clean commands that just remove Srcfiles from the given component (which it seems would also force a regeneration of Depends).

So can we have something like `./case.build --reset-srcfiles [component]` that calls something like `make clean-srcfiles-component` for the given component?


Discussion:
I think that this is a user knowledge issue - any attempt we make to automate it will be fraught with unintended consequences.    Even if we added this kind of command most users wouldn't look for it and wouldn't understand what it did when they found it.   
@jedwards4b sorry, I'm not sure I understand your concerns.

> any attempt we make to automate it will be fraught with unintended consequences

I understand your concerns with automating it. Please note that I'm not suggesting trying to automate it.

> Even if we added this kind of command most users wouldn't look for it 

Is that really a reason not to add it? By that argument, we should remove most of the tools in cime.

> and wouldn't understand what it did when they found it.

Isn't this just a documentation issue? Would you be open to adding this option if it were documented clearly?

The reason I'm running into this problem more now - and will continue to run into it more in the future - is that I am more and more using a workflow that greatly speeds up my development:

1. Use the same git clone for multiple branches, checking out the one I'm currently working on

2. Create a test case out of that clone, build it to completion

3. As I make changes to Fortran source code, rebuild that test case. The incremental rebuild tells me within seconds (rather than minutes) whether my changes compile.

The great thing about this workflow is that I can check out some other branch, make some small tweak, then do an incremental rebuild from the same test case, without having to set up a whole new test and wait minutes for the initial build. However, if the other branch has significant changes in what files are present, then I run into the problem listed in this issue. Sure, I can manually remove Srcfiles from the appropriate location, but particularly for CLM (where this is buried in a directory structure that's about 8 deep) this can take me a while to find, so a command like this would help me and others who want to do something similar.

@billsacks - I think what you are proposing sounds very reasonable. I
myself have wanted this type of capability. We can add additional examples
in the User's Guide along with the documentation in case.build that would
clarify this usage.

On Thu, Sep 14, 2017 at 1:12 PM, Bill Sacks <notifications@github.com>
wrote:

> @jedwards4b <https://github.com/jedwards4b> sorry, I'm not sure I
> understand your concerns.
>
> any attempt we make to automate it will be fraught with unintended
> consequences
>
> I understand your concerns with automating it. Please note that I'm not
> suggesting trying to automate it.
>
> Even if we added this kind of command most users wouldn't look for it
>
> Is that really a reason not to add it? By that argument, we should remove
> most of the tools in cime.
>
> and wouldn't understand what it did when they found it.
>
> Isn't this just a documentation issue? Would you be open to adding this
> option if it were documented clearly?
>
> The reason I'm running into this problem more now - and will continue to
> run into it more in the future - is that I am more and more using a
> workflow that greatly speeds up my development:
>
>    1.
>
>    Use the same git clone for multiple branches, checking out the one I'm
>    currently working on
>    2.
>
>    Create a test case out of that clone, build it to completion
>    3.
>
>    As I make changes to Fortran source code, rebuild that test case. The
>    incremental rebuild tells me within seconds (rather than minutes) whether
>    my changes compile.
>
> The great thing about this workflow is that I can check out some other
> branch, make some small tweak, then do an incremental rebuild from the same
> test case, without having to set up a whole new test and wait minutes for
> the initial build. However, if the other branch has significant changes in
> what files are present, then I run into the problem listed in this issue.
> Sure, I can manually remove Srcfiles from the appropriate location, but
> particularly for CLM (where this is buried in a directory structure that's
> about 8 deep) this can take me a while to find, so a command like this
> would help me and others who want to do something similar.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/1889#issuecomment-329581097>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxE_0NunbeIUj923fb0uAvHp8hG6M4ks5siXqigaJpZM4PXfYV>
> .
>


Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -311,11 +311,11 @@ def _build_model_thread(config_dir, compclass, compname, caseroot, libroot, bldr
     logger.info("{} built in {:f} seconds".format(compname, (t2 - t1)))
 
 ###############################################################################
-def _clean_impl(case, cleanlist, clean_all):
+def _clean_impl(case, cleanlist, clean_all, clean_depends):
 ###############################################################################
+    exeroot = os.path.abspath(case.get_value("EXEROOT"))
     if clean_all:
         # If cleanlist is empty just remove the bld directory
-        exeroot = os.path.abspath(case.get_value("EXEROOT"))
         expect(exeroot is not None,"No EXEROOT defined in case")
         if os.path.isdir(exeroot):
             logging.info("cleaning directory {}".format(exeroot))
@@ -327,7 +327,8 @@ def _clean_impl(case, cleanlist, clean_all):
             logging.warning("cleaning directory {}".format(sharedlibroot))
             shutil.rmtree(sharedlibroot)
     else:
-        expect(cleanlist is not None and len(cleanlist) > 0,"Empty cleanlist not expected")
+        expect((cleanlist is not None and len(cleanlist) > 0) or
+               (clean_depends is not None and len(clean_depends)),"Empty cleanlist not expected")
         debug           = case.get_value("DEBUG")
         use_esmf_lib    = case.get_value("USE_ESMF_LIB")
         build_threaded  = case.get_build_threaded()
@@ -345,10 +346,16 @@ def _clean_impl(case, cleanlist, clean_all):
         os.environ["CLM_CONFIG_OPTS"] = clm_config_opts  if clm_config_opts is not None else ""
 
         cmd = gmake + " -f " + os.path.join(casetools, "Makefile")
-        for item in cleanlist:
-            tcmd = cmd + " clean" + item
-            logger.info("calling {} ".format(tcmd))
-            run_cmd_no_fail(tcmd)
+        if cleanlist is not None:
+            for item in cleanlist:
+                tcmd = cmd + " clean" + item
+                logger.info("calling {} ".format(tcmd))
+                run_cmd_no_fail(tcmd)
+        else:
+            for item in clean_depends:
+                tcmd = cmd + " clean_depends" + item
+                logger.info("calling {} ".format(tcmd))
+                run_cmd_no_fail(tcmd)
 
     # unlink Locked files directory
     unlock_file("env_build.xml")
@@ -574,7 +581,7 @@ def case_build(caseroot, case, sharedlib_only=False, model_only=False, buildlist
     return run_and_log_case_status(functor, "case.build", caseroot=caseroot)
 
 ###############################################################################
-def clean(case, cleanlist=None, clean_all=False):
+def clean(case, cleanlist=None, clean_all=False, clean_depends=None):
 ###############################################################################
-    functor = lambda: _clean_impl(case, cleanlist, clean_all)
+    functor = lambda: _clean_impl(case, cleanlist, clean_all, clean_depends)
     return run_and_log_case_status(functor, "build.clean", caseroot=case.get_value("CASEROOT"))


Test patch: