Repository: ESMCI/cime

Issue:
Recursion in testmods_dirs (include_user_mods file)
In POP testing, we are having an issue where namelist changes (via `user_nl_pop`) are not being applied, and I'm not sure how to trace the problem. I'm also not sure of the best way to provide all the info needed in a readable manner, but we have (among others) the following directories in `components/pop/cime_config/testdefs/testmods_dirs/pop`:

```
cice_ecosys_dust_flux_driver/
cice/
ecosys_dust_flux_driver/
ecosys/
default/
```

And all but `default/` contain `include_user_mods` files:

```
$ cat cice_ecosys_dust_flux_driver/include_user_mods 
../cice
../ecosys_dust_flux_driver

$ cat cice/include_user_mods 
../default

$ cat ecosys_dust_flux_driver/include_user_mods 
../ecosys

$ cat ecosys/include_user_mods 
../default
```

Some of them also include `user_nl_pop` or `user_nl_cice` files:

```
$ find . -name user_nl_*
./cice/user_nl_cice
./ecosys_dust_flux_driver/user_nl_pop
./ecosys/user_nl_pop
./default/user_nl_pop
```

When we run a `pop-cice_ecosys_dust_flux_driver` test, we get the expected `user_nl_cice` but `user_nl_pop` matches the version in `default/` rather than containing the additional lines from `ecosys_dust_flux_driver/` and `ecosys/`

My current theory is that having two separate chains leading back to `default` is messing something up, because running just a `pop-ecosys_dust_flux_driver` test brings in all the expected `user_nl_pop` changes (but omits the `user_nl_cice` changes we would like to have).

---

A similar issue occurs with the `pop-ciso_abio_dic_dic14` test, which has the following chain of includes

```
$ cat ciso_abio_dic_dic14/include_user_mods 
../abio_dic_dic14
../ciso

$ cat abio_dic_dic14/include_user_mods 
../default

$ cat ciso/include_user_mods 
../ecosys

$ cat ecosys/include_user_mods 
../default
```

In this second case, only `default` and `ecosys` have `user_nl_pop` files, but the file from `ecosys/` is ignored. All of the `shell_commands` files (in `abio_dic_dic14/`, `ciso/`, and `default/`) are applied... it seems like a `user_nl` issue only.



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/user_mod_support.py b/utils/python/CIME/user_mod_support.py
--- a/utils/python/CIME/user_mod_support.py
+++ b/utils/python/CIME/user_mod_support.py
@@ -35,24 +35,9 @@ def apply_user_mods(caseroot, user_mods_path, ninst=None):
                 for comp_inst in xrange(1, ninst[comp]+1):
                     contents = newcontents
                     case_user_nl_inst = case_user_nl + "_%4.4d"%comp_inst
-                    logger.info("Pre-pending file %s"%case_user_nl_inst)
-                    if os.path.isfile(case_user_nl_inst):
-                        with open(case_user_nl_inst, "r") as fd:
-                            old_contents = fd.read()
-                            if old_contents.find(contents) == -1:
-                                contents = contents + old_contents
-                    with open(case_user_nl_inst, "w") as fd:
-                        fd.write(contents)
+                    update_user_nl_file(case_user_nl_inst, contents)
             else:
-                contents = newcontents
-                logger.info("Pre-pending file %s"%case_user_nl)
-                if os.path.isfile(case_user_nl):
-                    with open(case_user_nl, "r") as fd:
-                        old_contents = fd.read()
-                        if old_contents.find(contents) == -1:
-                            contents = contents + old_contents
-                with open(case_user_nl, "w") as fd:
-                    fd.write(contents)
+                update_user_nl_file(case_user_nl, newcontents)
 
         # update SourceMods in caseroot
         for root, _, files in os.walk(include_dir,followlinks=True,topdown=False):
@@ -85,6 +70,23 @@ def apply_user_mods(caseroot, user_mods_path, ninst=None):
             os.chmod(shell_command_file, 0777)
             run_cmd_no_fail(shell_command_file)
 
+def update_user_nl_file(case_user_nl, contents):
+    update_file = True
+    if os.path.isfile(case_user_nl):
+        with open(case_user_nl, "r") as fd:
+            old_contents = fd.read()
+        if old_contents.find(contents) == -1:
+            contents = contents + old_contents
+            update_file = True
+        else:
+            update_file = False
+    if update_file:
+        logger.info("Pre-pending file %s"%(case_user_nl))
+        with open(case_user_nl, "w") as fd:
+            fd.write(contents)
+
+
+
 def build_include_dirs_list(user_mods_path, include_dirs=None):
     '''
     If user_mods_path has a file "include_user_mods" read that


Test patch: