Repository: ESMCI/cime

Issue:
cloning: stream files are not copied
Using cime5.2.0-alpha.7

Custom stream files 
`user_datm.streams.txt.* `
are not copied when doing a ./create_clone


Discussion:
When I copy the stream files manually and with the following DATM user namelist: 
```
$ cat user_nl_datm
mapalgo = "nn","nn","nn","nn","nn"
```
preview_namelists fails to correctly produce datm_in: 
```
$ cat datm_in
&datm_nml
  decomp = "1d"
  factorfn = "null"
  force_prognostic_true = .false.
  iradsw = 1
  presaero = .true.
  restfilm = "undefined"
  restfils = "undefined"
  wiso_datm = .false.
/
&shr_strdata_nml
  datamode = "CLMNCEP"
  domainfile = "/glade/u/home/lvank/work/singlepoint/forcing/S9/domain.lnd.fv0.9x1.25_S9.nc"
  dtlimit = 1.5
  fillalgo = "nn"
  fillmask = "nomask"
  fillread = "NOT_SET"
  fillwrite = "NOT_SET"
  mapalgo = 5*"nn"
  mapmask = "nomask"
  mapread = "NOT_SET"
  mapwrite = "NOT_SET"
  streams =
  taxmode = "cycle"
  tintalgo = "linear"
  vectors = "null"
/
```
Example case on yellowstone:
`/glade/u/home/lvank/cesm/my_cases/bugfix_erai_1col/test_r205`
Because yellowstone is down I don't have enough information to reproduce your case and test this but I believe that the fix is:
```
index 4d9c6f8..0d52387 100644
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -971,9 +971,10 @@ class Case(object):
         newcase.create_caseroot(clone=True)
         newcase.flush(flushall=True)
 
-        # copy user_nl_files
+        # copy user_ files
         cloneroot = self._caseroot
-        files = glob.glob(cloneroot + '/user_nl_*')
+        files = glob.glob(cloneroot + '/user_*')
+        
         for item in files:
             shutil.copy(item, newcaseroot)
```
The datm_in that is being produced seems correct given the contents of
user_nl_datm (i.e. 5*"nn" is the same as nn","nn","nn","nn","nn" - just a
compressed notation).

Could you please clarify your workflow a bit more.

In streams.txt files should never be copied in a clone. If the file
generated is CaseDocs/streams.txt, this is overwritten everytime preview
namelist is called. The only think that should be coped in a clone is
$CASEROOT/user_streams.txt - which is a user generated file that is a copy
of streams.txt - but with the relevant user mods added.

What streams file did you copy over - and how did you copy them?





On Wed, Nov 30, 2016 at 8:09 AM, Leo van Kampenhout <
notifications@github.com> wrote:

> When I copy the stream files manually and with the following DATM user
> namelist:
>
> $ cat user_nl_datm
> mapalgo = "nn","nn","nn","nn","nn"
>
> preview_namelists fails to correctly produce datm_in:
>
> $ cat datm_in
> &datm_nml
>   decomp = "1d"
>   factorfn = "null"
>   force_prognostic_true = .false.
>   iradsw = 1
>   presaero = .true.
>   restfilm = "undefined"
>   restfils = "undefined"
>   wiso_datm = .false.
> /
> &shr_strdata_nml
>   datamode = "CLMNCEP"
>   domainfile = "/glade/u/home/lvank/work/singlepoint/forcing/S9/domain.lnd.fv0.9x1.25_S9.nc"
>   dtlimit = 1.5
>   fillalgo = "nn"
>   fillmask = "nomask"
>   fillread = "NOT_SET"
>   fillwrite = "NOT_SET"
>   mapalgo = 5*"nn"
>   mapmask = "nomask"
>   mapread = "NOT_SET"
>   mapwrite = "NOT_SET"
>   streams =
>   taxmode = "cycle"
>   tintalgo = "linear"
>   vectors = "null"
> /
>
> Example case on yellowstone:
> /glade/u/home/lvank/cesm/my_cases/bugfix_erai_1col/test_r205
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/850#issuecomment-263897011>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxEz3LVH_9uQRH0TGF-Ar1bxzNFbAtks5rDZGqgaJpZM4LARxS>
> .
>

Thanks both for the quick response. 

@jedwards4b I agree that will do the job. 

@mvertens The workflow in which I encountered these issues is in a single-column, offline CLM setup using single-point forcing files (hence the custom DATM stream files). I did some runs with CLM development version r187 before switching to r205 today, after which I noticed the different behaviour. Before, the `user_datm.streams.txt.*` were copied along with the `user_nl_*` files to the newly created case. 

Also, I still think there is something fishy with creating the datm_in in combination with  `user_datm.streams.txt.*` files, because when I remove them I get a completely different file. To name a few differences ( I can't log into Yellowstone either):   `dtlimit = 1.5` has also the 5* multiplier, and the `streams =` entry contains values. 

I have a workaround, which is copying the contents of a valid datm_in file into `user_nl_datm`. 
> What streams file did you copy over - and how did you copy them?

I don't know streams.txt or user_streams.txt, I only use user_datm.streams.txt.Precip and the like. 
I was just using streams.txt as a generic name. There is no such file.

On Wed, Nov 30, 2016 at 1:46 PM, Leo van Kampenhout <
notifications@github.com> wrote:

> What streams file did you copy over - and how did you copy them?
> I don't know streams.txt or user_streams.txt, I only use
> user_datm.streams.txt.Precip and the like.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/850#issuecomment-263990235>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxE9Ur_ZvZpAW8PKNtUTJrjq6fOHtCks5rDeCggaJpZM4LARxS>
> .
>


Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -971,9 +971,10 @@ def create_clone(self, newcase, keepexe=False, mach_dir=None, project=None):
         newcase.create_caseroot(clone=True)
         newcase.flush(flushall=True)
 
-        # copy user_nl_files
+        # copy user_ files
         cloneroot = self._caseroot
-        files = glob.glob(cloneroot + '/user_nl_*')
+        files = glob.glob(cloneroot + '/user_*')
+        
         for item in files:
             shutil.copy(item, newcaseroot)
 


Test patch: