Repository: ESMCI/cime

Issue:
create_newcase needs a non-interactive mode
Some ACME users of the run_acme script, like @singhbalwinder , have reported indefinite hangs in create_newcase because, unknown to them, it was waiting for an answer to this question:

Directory %s already exists, (r)eplace, (a)bort, or (u)se existing?

We need create_newcase to have a non-interactive flag that will automatically answer "u" to the above question and then make run_acme pass this flag.


Discussion:
If someone is using a script, shouldn't the script handle this explicitly?
```
CASE=/foo/bar
if [ -d "${CASE}" ]; then
  rm -rf ${CASE}
fi
./create_newcase --case ${CASE} ...
```
I'm afraid that if we put an `rm -rf` switch into create_newcase, we will get an angry support issue every time some hurried user types:
`./create_newcase --overwrite --compset F --grid ne16_ne16 --case /`

For some reason, I ran into this issue only on some machines. Following is my workflow when I run into this issue:

1. Run a scripts (lets call it "run_script") which cd into cime/scripts and run create_newcase
2. "run_script" also builds the model and submit a run. So we get a case directory, a bld directory and a run directory after this whole script executes.
3. Modify a F90 file or add/modify a namelist parameter in my run_script.
4. Run the "run_script" again.

On some machines, run_script would just freeze at create_newcase step (at step 4 above) without any explicit message. To further explore this, I went into cime/scripts and issued "create_newcase" command there and then I got a message like:
```Directory /dtemp/sing201/csmruns/acme_def_run_02232017/bld already exists, (r)eplace, (a)bort, or (u)se existing?```

At this point I realized that I have to tell the script to use the existing bld directory. This message didn't appear at step 4 above in my workflow.

Is "run_script" designed to be used that way?  I mean both for initially creating/building/running a case and re-doing those last two steps?  So the idea is you just type "run_script" over and over no matter where you are in the config-build-run workflow?
That is how I have been running it for as long as I have run the models (CESM/ACME). I do this so that I can capture all my changes (configuration, namelist changes etc.) in one place. Other option is to make changes through the case root directory (which is also fine) but in that case I may forget what I changed in the case root directory at a later date. Having said that, I understand there may be some changes which I have to make in the case root directory.

If I keep all my changes in one script (e.g. run_script) and I know the model hash, I can reliably reproduce the model run.
  
I use this method as well but usually have explicit logic to decide what to do if the case already exists.
However, I am a bit unclear on one issue in your description.

Are you running this script interactively and still not seeing the 'already exists' prompt?
Yes, I think that is the issue. I am running "run_script" interactively and not seeing this prompt. When I run "create_newcase" directly from within the source code (cime/scripts), then I see this prompt. 


Also, "create_newcase" doesn't complain about the case root directory (I forcefully remove that in my "run_script"), it complains about the existence of "bld" directory.
This is almost sounding like a bug in `create_newcase`.
It sounds like your case has the case directory in one location and the bld directory somewhere else (e.g., in scratch, not under your case directory). <= Do I have that right?
Something seems fishy. If your script is not too long, can you post it (at least the bit up through the `create_newcase` call)?
Yes. My case root directory is in home but my bld and run directories are in something like scratch space (/lustre/$LOGNAME in this case).Here is the script I am using up until create_newcase:
```
#!/bin/csh

#set echo verbose
set run_time = 00:10:00
set queue    = slurm
set ntasks   = 24
set nthrds   = 1

set met_data_yr = '0001'
set met_data_mn = '01'
set met_data_dy = '02'


## ====================================================================
#   Set CASE related variables
## ====================================================================
setenv CASE      acme_def_run_02072017
setenv MACH      sooty
setenv COMPILER  intel
setenv COMPSET   FC5AV1C-04P2
setenv RES       ne16_ne16
setenv MAKE_J    24
setenv mods      'none' #$locdir/mods_$CASE #'none' for no mods directory

## ====================================================================
#   Define directories
## ====================================================================
setenv PTMP      /lustre/$LOGNAME/csmruns
setenv SRCROOT     ~sing201/acme/srccode/02072017/ACME
setenv CASEROOT  $PTMP/CSM/CMIP5/$CASE
setenv RUNDIR    {$PTMP}/$CASE/run
setenv BLDDIR    {$PTMP}/$CASE/bld

## ====================================================================
#   create new case
## ====================================================================
rm -rf $CASEROOT
cd  $SRCROOT/cime/scripts
echo '==== GIT STATUS OF THE SRC ===='
git status
git rev-parse HEAD
echo '==== GIT STATUS ENDS ===='
module rm python
module load python/2.7.8
which python
./create_newcase -case $CASEROOT -mach $MACH -res $RES -compset $COMPSET -compiler $COMPILER --project none|| echo 'create_newcase failed' && exit -1
```
I don't believe that the default should be to use the existing run directory - either the default should be to remove and replace or as @gold2718 suggested the script should explicitly remove the directory before calling create_newcase.
@jedwards4b : Wouldn't that be very different from what is assumed "default" now? I think most users now think that launching create_newcase would not replace or remove the build. If we are going to change this, we should let the community know of this change.
I think that most users expect create_newcase is creating a newcase and thus would assume that they are starting from scratch.   Using an existing build with a new create_newcase command is, I think unusual - if for example a user creates an F case called foo, then goes back later and creates a B case called foo inheriting the same bld would be a problem.
I agree, in that case it would be a problem. I have never done that so I don't know at which step the build system will complain in this scenario.

 I am talking about a scenario where everything stays the same but we issue "create_newcase" twice. In this case, usually a user (at least me) assumes that this case will use the same bld directory.
So how about instead of just defaulting to 'u' we add a new command line option to create_newcase which allows the user to choose r, a or u.
@jedwards4b seems reasonable
I think this functionality already exists (in file `cime/utils/python/CIME/case.py`) but the issue is that this prompt doesn't show up on some machines. On other machines it just defaults to `u` behavior. 
@jedwards4b, if we are going to have automatic 'r' behavior, can we at least make sure it is a CIME case/bld/run directory before wiping it? The possibilities for our script to destroy user files is quite large and typical users do not react kindly to "You shouldn't have done that".
@singhbalwinder, Still trying to understand the behavior. Are these true statements?

- The missing prompt only shows up in regards to the bld and/or run directories (because you explicitly remove the CASEROOT).
- Some machines happily try to reuse the bld directory while others hang.
- You never spoke with Russian officials about presidential politics

I'm asking because I don't think `create_newcase` creates the bld or run directories so is that really where the hang is?
@singhbalwinder, 
A work around would be for you to remove the bld and run directories just before you remove the CASEROOT.
```
(cd $CASEROOT; rm -rf `./xmlquery --value EXEROOT`)
(cd $CASEROOT; rm -rf `./xmlquery --value RUNDIR`)
```
@gold2718 I think that you should reread the background, if I understand correctly he would like to leave the bld directory in place and reuse it for his workflow.
@jedwards4b, yes but you said: 

> So how about instead of just defaulting to 'u' we add a new command line option to create_newcase which allows the user to choose r, a or u.

which means you proposed adding a feature which would allow create_newcase to delete any directory without prompting the user. If you see my 'accidentally hit the return key too soon' example from above (second comment), I think this is dangerous.
@gold2718 : Yes, all your statements are true including the third one :grinning: . On some machines create_newcase hangs. On those machines, if I cd into cime/scripts and launch create_newcase with the same arguments, I see this prompt:
` Directory /dtemp/sing201/csmruns/acme_def_run_02232017/bld already exists, (r)eplace, (a)bort, or (u)se existing? `

Now, on these machines, if I go back and run my "run_script" and type `u` when it hangs, it moves along happily (which is kind of obvious now).

I don't have a 'preferred' workflow but I always acted under the assumption that `u` is the default behavior.

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -530,7 +530,7 @@ def configure(self, compset_name, grid_name, machine_name=None,
                   project=None, pecount=None, compiler=None, mpilib=None,
                   user_compset=False, pesfile=None,
                   user_grid=False, gridfile=None, ninst=1, test=False,
-                  walltime=None, queue=None, output_root=None, run_unsupported=False):
+                  walltime=None, queue=None, output_root=None, run_unsupported=False, answer=None):
 
         #--------------------------------------------
         # compset, pesfile, and compset components
@@ -752,7 +752,11 @@ def configure(self, compset_name, grid_name, machine_name=None,
             logging.debug("wdir is %s"%wdir)
             if os.path.exists(wdir):
                 expect(not test, "Directory %s already exists, aborting test"% wdir)
-                response = raw_input("\nDirectory %s already exists, (r)eplace, (a)bort, or (u)se existing?"% wdir)
+                if answer is None:
+                    response = raw_input("\nDirectory %s already exists, (r)eplace, (a)bort, or (u)se existing?"% wdir)
+                else:
+                    response = answer
+
                 if response.startswith("r"):
                     shutil.rmtree(wdir)
                 else:


Test patch: