Repository: ESMCI/cime

Issue:
problem rebuilding if case.build fails after a successful first build
I am encountering the following failure.
1) I invoke case.build successfully
2) I realize I want to change a file - and I modify a file (say in CLM)
3) I run case.build again - and there is a compilation failure
4) I fix the file so that I expect it will now compile - and I rerun case.build. I then get the following:
```
Setting build complete to False
ERROR:
ERROR env_build HAS CHANGED
  A manual clean of your obj directories is required
  You should execute the following:
    ./case.build --clean-all
```
I know that I don't want to run case.build --clean-all since everything should work except for the file that I had changed. It turns out that to fix this I need to run the following two steps:
```
rm LockedFiles/env_build.xml
./xmlchange BUILD_STATUS=0
```
This is clearly a problem that users would run into when making changes to a case.
 


Discussion:
I just had this happen on Hobart but don't know what changed. The sequence was:

1. `create_newcase`, `case.setup`, `case.build`, `case.submit`, `case.run` (run crashed)
2. Edit Fortran file
3. `case.build` (build crashed)
4. Fix error in edit
5. `case.build`

```
Building case in directory /scratch/cluster/goldy/QPC6_mg16
sharedlib_only is False
model_only is False
File /scratch/cluster/goldy/QPC6_mg16/LockedFiles/env_build.xml has been modified
  found difference in BUILD_COMPLETE : case False locked True
Setting build complete to False
ERROR: 
ERROR env_build HAS CHANGED
  A manual clean of your obj directories is required
  You should execute the following:
    ./case.build --clean-all
```
Could it be some sort of race condition that locks up the process?
Maybe it is the sequence of complete build, failed build, build try that triggers things. My diff:
```
hobart: diff env_build.xml LockedFiles/
264c264
<     <entry id="BUILD_COMPLETE" value="FALSE">
---
>     <entry id="BUILD_COMPLETE" value="TRUE">
289c289
<     <entry id="BUILD_STATUS" value="1">
---
>     <entry id="BUILD_STATUS" value="0">
```
I am closing this issue since noone seems to have come up with a reproducer. 
I did come up with a reproducer which is the comment above from about Oct. 22. 
To restate it:

1. Build the model
2. Make a code change that will not compile
3. Try another build (build fails)
4. Make another change and try a build again (triggers issue)

Does this not trigger the error for you?
No it does not, perhaps I am not making the right change, but I tried several times. 
Hmm. Well, all I can say is that I have been making bad code changes in CAM (I'm particularly talented at that) and getting failed compiles (`./case.build`). Then, whenever I try to `./case.build` again, I get the message. Could it depend on the component being modified?
Can you give me instructions to reproduce in which you tell me exactly how to generate this error in a reproducible fashion?  Otherwise I would again request that you close it.   
I have had the exact same problem as @gold2718 - I'll try to duplicate this
problem on cime master again.

On Thu, Nov 2, 2017 at 2:22 PM, goldy <notifications@github.com> wrote:

> Hmm. Well, all I can say is that I have been making bad code changes in
> CAM (I'm particularly talented at that) and getting failed compiles (
> ./case.build). Then, whenever I try to ./case.build again, I get the
> message. Could it depend on the component being modified?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/1971#issuecomment-341514513>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxE1u4Ect5mMgiZrX618Al5Ofgyo3_ks5syghOgaJpZM4P9Cx7>
> .
>

Since no reproducer has been presented for this issue, I still think that it should be closed.  I have removed the critical tag.
Reproducer using cime-only clone:
```
./create_newcase --case /scratch/cluster/${USER}/A_mg17 --compset A --res f19_f19_mg17 --compiler nag --run-unsupported
cd /scratch/cluster/${USER}/A_mg17
./xmlchange DEBUG=TRUE
./case.setup
./case.build
sed -i -e 's/SHR_KIND_R8/SHR_KIND_FOODIDOO/' `./xmlquery --value CIMEROOT`/src/drivers/mct/main/cime_comp_mod.F90
./case.build
sed -i -e 's/SHR_KIND_FOODIDOO/SHR_KIND_R8/' `./xmlquery --value CIMEROOT`/src/drivers/mct/main/cime_comp_mod.F90
./case.build
```

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -1330,8 +1330,6 @@ def set_file(self, xmlfile):
         gfile = GenericXML(infile=xmlfile)
         ftype = gfile.get_id()
 
-        self.flush(flushall=True)
-
         logger.warning("setting case file to {}".format(xmlfile))
         components = self.get_value("COMP_CLASSES")
         new_env_file = None
diff --git a/scripts/lib/CIME/check_lockedfiles.py b/scripts/lib/CIME/check_lockedfiles.py
--- a/scripts/lib/CIME/check_lockedfiles.py
+++ b/scripts/lib/CIME/check_lockedfiles.py
@@ -111,10 +111,12 @@ def check_lockedfiles(case):
                 continue
             diffs = f1obj.compare_xml(f2obj)
             if diffs:
+
                 logging.warning("File {} has been modified".format(lfile))
                 for key in diffs.keys():
-                    print("  found difference in {} : case {} locked {}"
-                          .format(key, repr(diffs[key][0]), repr(diffs[key][1])))
+                    if key != "BUILD_COMPLETE":
+                        print("  found difference in {} : case {} locked {}"
+                              .format(key, repr(diffs[key][0]), repr(diffs[key][1])))
 
                 if objname == "env_mach_pes":
                     expect(False, "Invoke case.setup --reset ")
@@ -128,8 +130,9 @@ def check_lockedfiles(case):
                         case.set_value("BUILD_STATUS", 2)
                         logging.critical("Changing PIO_VERSION requires running "
                                          "case.build --clean-all and rebuilding")
-                    else:
+                    elif key != "BUILD_COMPLETE":
                         case.set_value("BUILD_STATUS", 1)
+                    f2obj.set_value("BUILD_COMPLETE", False)
                 elif objname == "env_batch":
                     expect(False, "Batch configuration has changed, please run case.setup --reset")
                 else:


Test patch: