Repository: ESMCI/cime

Issue:
check_input_data check passed for directory, does not confirm if listing is a file
check_input_data appears only to check for the existence of elements listed in the input_data_list, and not whether or not they are files. For example, check_input_data will pass the following:

Buildconf/cice.input_data_list
grid_file = /work/02441/bcash/stampede2/UFS/ufs_inputdata/
kmt_file = /work/02441/bcash/stampede2/UFS/ufs_inputdata/

The subsequent run will fail, however, because grid_file and kmt_file are required.

If I understand the code correctly (which I probably do not) the issue is that check_input_data.py is using os.path.exists in lines 342 and 358 to perform the existence check, but os.path.exists does not distinguish between files and directories. 


Discussion:
Assigning to @WesCoomber since this should be a pretty easy fix.
@benjamin-cash Hi I am having some trouble recreating this error. I created a new test `./create_test SMS.f19_f19.A`  . Then I wiped away some of the dice.input_data_list and did a `case.submit` expecting a run error. There is no run error because it looks like the case.submit regenerates all of the *.input_data_list files in the case/Buildconf/ directory. Could you provide some more information on the steps of how you are encountering the run error?
@WesCoomber - That doesn't quite sound like the problem I am experiencing. The issue I have is that, for whatever reason,  after ./case.build was run BuildConf/cice.input_data_list had incorrect values for two key files. Specifically, they pointed to directories and not files. 

check_input_data did not produce an error because,  as far as I can tell, it is using os.path.exists and the directory does in fact exist. But the application needed a data file, not a directory, so it failed. Does that make sense?
@benjamin-cash Could you let me know which case is creating incorrect values in BuildConf/cice.input_data_list? I am trying to test a fix but the cases I am using automatically generate working *.input_data_list files when I do a ./case.build and when I do a ./case.submit. So that means I am having trouble testing the fix.
@WesCoomber - If you edit the *.input_data_list file to point to a directory and then just run ./check_input_data independently of ./case.submit does it pass the files?
@benjamin-cash 
Ok that makes sense. I made the fix, and modified dice.input_data_list in case/BuildConf/ to point to a directory. `./check_input_data` correctly detects the missing file now.

```
mappy@<~/acme/scratch/SMS.f19_f19.A.mappy_gnu.20200810_181741_fgcsb7>% ./check_input_data
Loading input file list: 'Buildconf/2dice.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Loading input file list: 'Buildconf/dice.input_data_list'
Model dice missing file domain1 = '/sems-data-store/ACME/inputdata/ice/dice7/'
Loading input file list: 'Buildconf/docn.input_data_list'
Loading input file list: 'Buildconf/drof.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
```
@WesCoomber , your fix for this broke E3SM. The errors look like this on mappy:

```
% ./create_test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu
No project info available
create_test will do up to 1 tasks simultaneously
create_test will use up to 80 cores simultaneously
Creating test directory /ascldap/users/jgfouca/acme/scratch/SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu.20200819_122804_cbd7eo
RUNNING TESTS:
  SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu
Starting CREATE_NEWCASE for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu with 1 procs
Finished CREATE_NEWCASE for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu in 1.314028 seconds (PASS)
Starting XML for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu with 1 procs
Finished XML for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu in 0.226263 seconds (PASS)
Starting SETUP for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu with 1 procs
Finished SETUP for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu in 2.052240 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu in 27.392404 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu in 34.559633 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu with 16 procs
Finished RUN for test SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu in 3.336394 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /ascldap/users/jgfouca/acme/scratch/SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu.20200819_122804_cbd7eo
    Errors were:
        wget failed with output:  and errput File `/sems-data-store/ACME/inputdata/atm/cam/chem/trop_mozart/ub' already there; not retrieving.
        
            Cannot download file since it lives outside of the input_data_root '/sems-data-store/ACME/inputdata'
        svn export failed with output:  and errput svn: E155000: Destination directory exists; please remove the directory or use --force to overwrite
        svn: E155000: '/sems-data-store/ACME/inputdata/atm/cam/chem/trop_mozart/ub' already exists
        
            Cannot download file since it lives outside of the input_data_root '/sems-data-store/ACME/inputdata'
        ERROR: Could not find all inputdata on any server

At test-scheduler close, state is:
FAIL SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu (phase RUN)
    Case dir: /ascldap/users/jgfouca/acme/scratch/SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu.20200819_122804_cbd7eo
test-scheduler took 69.4313359261 seconds
```

Please rethink your fix for this issue. Also, let's see if we can add a test to scripts_regression_tests so that we don't have this issue again. E3SM must be doing something with check_input_data that is not currently being tested.
Note that /sems-data-store/ACME/inputdata/atm/cam/chem/trop_mozart/ub is a directory and not a file and so this is probably an error in the namelist_defaults.xml file in cam.
Even if that is the case, the error message printed here should be more clear.

@benjamin-cash I double checked and "cam.input_data_list" is originally intended to have data paths to directories And file paths to files.

For example, in a successful case test `acme/scratch/SMS_D_Ln5.ne4_ne4.FC5AV1C-L.mappy_gnu.20200821_204136_ox369z/Buildconf/cam.input_data_list`

I see at least 3 entries that specify a directory path
```

tracer_cnst_datapath = /sems-data-store/ACME/inputdata/atm/cam/chem/trop_mozart_aero/oxid
linoz_data_path = /sems-data-store/ACME/inputdata/atm/cam/chem/trop_mozart/ub
mam_mom_datapath = /sems-data-store/ACME/inputdata/atm/cam/chem/trop_mam/marine_BGC/
```

_That means that the original issue raised here is actually already solved. The check_input_data tool does not error out on directory paths because some input_data is correctly directory paths._ 

Let me know if that makes sense.
It makes sense, although in my case it remains an issue because it allowed an incorrectly specified input_data_list to go through and the model crashed. It would be much more useful as a checker if there was some way to specify if something is supposed to be a file or a directory and then check_input_data verified that, rather than just if the path exists.  
@WesCoomber if cam.input_data_list has directory paths, those could all be mistakes.  For that to be allowed, there should be some logic in check_input_data to check every file in that path against the files in the same path on the input data servers and download missing files.

Resolving patch:
diff --git a/scripts/lib/CIME/case/check_input_data.py b/scripts/lib/CIME/case/check_input_data.py
--- a/scripts/lib/CIME/case/check_input_data.py
+++ b/scripts/lib/CIME/case/check_input_data.py
@@ -316,8 +316,15 @@ def check_input_data(case, protocol="svn", address=None, input_data_root=None, d
             if (line and not line.startswith("#")):
                 tokens = line.split('=')
                 description, full_path = tokens[0].strip(), tokens[1].strip()
-                if description.endswith('datapath') or description.endswith('data_path'):
+                if description.endswith('datapath') or description.endswith('data_path') or full_path.endswith('/dev/null'):
                     continue
+                if description.endswith('file') or description.endswith('filename'):
+                    # There are required input data with key, or 'description' entries
+                    # that specify in their names whether they are files or filenames
+                    # rather than 'datapath's or 'data_path's so we check to make sure
+                    # the input data list has correct non-path values for input files.
+                    # This check happens whether or not a file already exists locally. 
+                    expect((not full_path.endswith(os.sep)), "Unsupported directory path in input_data_list named {}. Line entry is '{} = {}'.".format(data_list_file, description, full_path))
                 if(full_path):
                     # expand xml variables
                     full_path = case.get_resolved_value(full_path)
@@ -333,7 +340,6 @@ def check_input_data(case, protocol="svn", address=None, input_data_root=None, d
                         if ic_filepath:
                             rel_path  = full_path.replace(input_ic_root, ic_filepath)
                         use_ic_path = True
-
                     model = os.path.basename(data_list_file).split('.')[0]
 
                     if ("/" in rel_path and rel_path == full_path and not full_path.startswith('unknown')):


Test patch: