Repository: ESMCI/cime

Issue:
check_input_data not supporting input files outside of input data repo
@erichlf has asked for support for the following use case:
* Create a partition file in his home directory
* Point namelist files to use this partition file
* check_input_data should find this file but it doesn't because it insists on looking in the input data repo


Discussion:
By what mechanism is he doing step 2?  It should be through user_nl files. 
In user_nl_mpas{o,cice} I set 
config_block_decomp_file_prefix
which results in the proper variable being set in BuildConf/mpas-{o,cice}conf/mpas-{o,cice}_in. However, the graph24 variable is not correct in BuildConf/mpas-{o,cice}.input_data_list.
As a work-around I can do a ./case_setup and then set graph24 manually.
@erichlf okay - I would expect this to work - are you setting a fully qualified path or a relative path?
I suspect the mpas buildnml scripts in the cime_config dir. 
@jedwards4b I set the full path.
@erichlf I just tested this by hand. If you manually add a file to the $model.input_data_list, it works fine (uses the path you provided).

If you set the full path in user_nl_*, you do get some odd behavior that I will fix.

Resolving patch:
diff --git a/utils/python/CIME/check_input_data.py b/utils/python/CIME/check_input_data.py
--- a/utils/python/CIME/check_input_data.py
+++ b/utils/python/CIME/check_input_data.py
@@ -148,32 +148,45 @@ def check_input_data(case, svn_loc=None, input_data_root=None, data_list_dir="Bu
                     # expand xml variables
                     full_path = case.get_resolved_value(full_path)
                     rel_path  = full_path.replace(input_data_root, "")
+                    model = os.path.basename(data_list_file).split('.')[0]
 
-                    # There are some special values of rel_path that
-                    # we need to ignore - some of the component models
-                    # set things like 'NULL' or 'same_as_TS' -
-                    # basically if rel_path does not contain '/' (a
-                    # directory tree) you can assume it's a special
-                    # value and ignore it (perhaps with a warning)
-                    if ("/" in rel_path and not os.path.exists(full_path)):
-                        model = os.path.basename(data_list_file).split('.')[0]
-                        logging.warning("Model %s missing file %s = '%s'" % (model,description,full_path))
-
-                        if (download):
-                            success = download_if_in_repo(svn_loc, input_data_root, rel_path)
-                            if (not success):
-                                # If ACME, try CESM repo as backup
-                                if (get_model() == "acme" and svn_loc != SVN_LOCS["cesm"]):
-                                    success = download_if_in_repo(SVN_LOCS["cesm"], input_data_root, rel_path)
-                                    if (not success):
-                                        no_files_missing = False
-                                else:
-                                    no_files_missing = False
-                        # if not download
-                        else:
+                    if ("/" in rel_path and rel_path == full_path):
+                        # User pointing to a file outside of input_data_root, we cannot determine
+                        # rel_path, and so cannot download the file. If it already exists, we can
+                        # proceed
+                        if not os.path.exists(full_path):
+                            logging.warning("  Model %s missing file %s = '%s'" % (model, description, full_path))
+                            if download:
+                                logging.warning("    Cannot download file since it lives outside of the input_data_root '%s'" % input_data_root)
                             no_files_missing = False
+                        else:
+                            logging.info("  Found input file: '%s'" % full_path)
+
                     else:
-                        logging.debug("Already had input file: '%s'" % full_path)
+                        # There are some special values of rel_path that
+                        # we need to ignore - some of the component models
+                        # set things like 'NULL' or 'same_as_TS' -
+                        # basically if rel_path does not contain '/' (a
+                        # directory tree) you can assume it's a special
+                        # value and ignore it (perhaps with a warning)
+                        if ("/" in rel_path and not os.path.exists(full_path)):
+                            logging.warning("  Model %s missing file %s = '%s'" % (model,description,full_path))
+
+                            if (download):
+                                success = download_if_in_repo(svn_loc, input_data_root, rel_path)
+                                if (not success):
+                                    # If ACME, try CESM repo as backup
+                                    if (get_model() == "acme" and svn_loc != SVN_LOCS["cesm"]):
+                                        success = download_if_in_repo(SVN_LOCS["cesm"], input_data_root, rel_path)
+                                        if (not success):
+                                            no_files_missing = False
+                                    else:
+                                        no_files_missing = False
+                            # if not download
+                            else:
+                                no_files_missing = False
+                        else:
+                            logging.info("  Already had input file: '%s'" % full_path)
 
                 else:
                     model = os.path.basename(data_list_file).split('.')[0]


Test patch: