Repository: ESMCI/cime

Issue:
gftp sometimes creates empty file if inputdata file isn't on server
I did some testing of a new initial condition file on `cheyenne`, and then ran `rimport` to add it to the subversion repository. I then jumped over to `hobart` and tried to run a test with the new file... but the test failed and when I ran `ls -l` I realized the new file was empty

```
$ ls -l /fs/cgd/csm/inputdata/ocn/pop/gx3v7/ic/ecosys_jan_IC_gx3v7_20190712.nc
-rw-rw-r-- 1 mlevy cgdcsm 0 Jul 15 11:06 /fs/cgd/csm/inputdata/ocn/pop/gx3v7/ic/ecosys_jan_IC_gx3v7_20190712.nc
```

I asked @jedwards4b about this, and he said that the `gftp` protocol was leaving the empty file. Looking at the output from `case.submit`, I realized that `gftp` was returning a non-success status code:

```
FAIL: GridFTP repo 'ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/' does not have file 'ocn/pop/gx3v7/ic/ecosys_jan_IC_gx3v7_20190712.nc' error=error: globus_ftp_client: the server responded with an error
500 500-Command failed. : globus_l_gfs_file_open failed.
500-globus_xio: Unable to open file /cesm/inputdata/ocn/pop/gx3v7/ic/ecosys_jan_IC_gx3v7_20190712.nc
500-globus_xio: System error in open: No such file or directory
500-globus_xio: A system call failed: No such file or directory
500 End.
```

Maybe `cime` could run `rm -f` on the file if `gftp` doesn't think it successfully pulled it down? 

Two notes:

1. I worked around this issue by promoting subversion in `config_inputdata.xml`, and also could have just waited for a `cron` job to add the file to the `gftp` server, so this isn't a critical issue
2. @jedwards4b couldn't reproduce the problem, so maybe it isn't `gftp`'s fault? Nothing in standard out indicates that `wget` or `ftp` were used to get the file, so I assumed the empty file was created by `gftp`


Discussion:
Here's even more output (running `check_input_data --download` with the verbose flag enabled)

```
$ ./check_input_data --download -v
07-15 12:18 CIME.XML.env_mach_specific INFO     Setting resource.RLIMIT_STACK to -1 from (-1, -1)
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/cpl.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/pop.input_data_list'
07-15 12:18 CIME.case.check_input_data WARNING    Model pop missing file init_ecosys_init_file = '/fs/cgd/csm/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/drof.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/ww3.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/datm.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/dice.input_data_list'
07-15 12:18 CIME.case.check_input_data INFO     Using protocol gftp with user  and passwd
07-15 12:18 root         INFO     Trying to download file: '../inputdata_checksum.dat' to path '/scratch/cluster/mlevy/SMS_Ld1.T62_g17.C1850ECO.hobart_nag.pop-ecosys_add_cocco.20190715_114315_p5pvme/run/inputdata_checksum.dat.raw' using GridFTP protocol.
07-15 12:18 CIME.case.check_input_data INFO     Using protocol wget with user anonymous and passwd user@example.edu
07-15 12:18 root         INFO     Trying to download file: '../inputdata_checksum.dat' to path '/scratch/cluster/mlevy/SMS_Ld1.T62_g17.C1850ECO.hobart_nag.pop-ecosys_add_cocco.20190715_114315_p5pvme/run/inputdata_checksum.dat.raw' using WGET protocol.
07-15 12:18 root         INFO     SUCCESS

07-15 12:18 CIME.case.check_input_data INFO     Using protocol ftp with user anonymous and passwd user@example.edu
07-15 12:18 CIME.Servers.ftp INFO     server address ftp.cgd.ucar.edu root path cesm/inputdata
07-15 12:18 root         INFO     Trying to download file: '../inputdata_checksum.dat' to path '/scratch/cluster/mlevy/SMS_Ld1.T62_g17.C1850ECO.hobart_nag.pop-ecosys_add_cocco.20190715_114315_p5pvme/run/inputdata_checksum.dat.raw' using FTP protocol.
07-15 12:18 CIME.case.check_input_data INFO     Using protocol svn with user  and passwd
07-15 12:18 CIME.case.check_input_data WARNING  Client protocol None not enabled
07-15 12:18 CIME.case.check_input_data INFO     Checking server ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/ with protocol gftp
07-15 12:18 CIME.XML.env_mach_specific INFO     Setting resource.RLIMIT_STACK to -1 from (-1, -1)
07-15 12:18 CIME.case.check_input_data INFO     Using protocol gftp with user  and passwd
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/cpl.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/pop.input_data_list'
07-15 12:18 CIME.case.check_input_data WARNING    Model pop missing file init_ecosys_init_file = '/fs/cgd/csm/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc'
07-15 12:18 root         INFO     Trying to download file: 'ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc' to path '/fs/cgd/csm/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc' using GridFTP protocol.
07-15 12:18 root         WARNING  FAIL: GridFTP repo 'ftp://gridanon.cgd.ucar.edu:2811/cesm/inputdata/' does not have file 'ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc' error=error: globus_ftp_client: the server responded with an error
500 500-Command failed. : globus_l_gfs_file_open failed.
500-globus_xio: Unable to open file /cesm/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc
500-globus_xio: System error in open: No such file or directory
500-globus_xio: A system call failed: No such file or directory
500 End.

07-15 12:18 root         INFO     Loading input file list: 'Buildconf/drof.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/ww3.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/datm.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/dice.input_data_list'
07-15 12:18 CIME.case.check_input_data INFO     Checking server ftp://ftp.cgd.ucar.edu/cesm/inputdata with protocol wget
07-15 12:18 CIME.XML.env_mach_specific INFO     Setting resource.RLIMIT_STACK to -1 from (-1, -1)
07-15 12:18 CIME.case.check_input_data INFO     Using protocol wget with user anonymous and passwd user@example.edu
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/cpl.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/pop.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/drof.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/ww3.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/datm.input_data_list'
07-15 12:18 root         INFO     Loading input file list: 'Buildconf/dice.input_data_list'
$ ls -l /fs/cgd/csm/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc
-rw-rw-r-- 1 mlevy cgdcsm 0 Jul 15 12:18 /fs/cgd/csm/inputdata/ocn/pop/gx1v6/ic/ecosys_jan_IC_gx1v6_20190712.nc
$
```

It's still unclear to me where the problem is actually occurring if it's not in the `gftp` step
I just ran into a similar problem. As with @mnlevy1981 I added a new file to the inputdata repo from cheyenne, and when I ran tests on hobart, they failed because the file on hobart had 0 size. Relevant output from TestStatus.log below:

```
Using protocol wget with user anonymous and passwd user@example.edu
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/clm.input_data_list'
Trying to download file: 'lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc' to path '/fs/cgd/csm/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc' using WGET protocol.
Loading input file list: 'Buildconf/cism.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Checking server ftp.cgd.ucar.edu/cesm/inputdata with protocol ftp
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol ftp with user anonymous and passwd user@example.edu
server address ftp.cgd.ucar.edu root path cesm/inputdata
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/clm.input_data_list'
Trying to download file: 'lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc' to path '/fs/cgd/csm/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc' using FTP protocol.
Loading input file list: 'Buildconf/cism.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Checking server https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata with protocol svn
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol svn with user  and passwd
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/clm.input_data_list'
Loading input file list: 'Buildconf/cism.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Check case OK
Submitting job script qsub -q medium -l walltime=00:20:00 -v ARGS_FOR_SCRIPT='--skip-preview-namelist' .case.test
Submitted job id is 189207.hobart.cgd.ucar.edu
Submitted job case.test with id 189207.hobart.cgd.ucar.edu
Model clm missing file finidat = '/fs/cgd/csm/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc'
Client protocol gftp not enabled
Client protocol None not enabled
Client protocol gftp not enabled
  Model clm missing file finidat = '/fs/cgd/csm/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc'
wget failed with output:  and errput --2019-07-18 20:30:43--  ftp://ftp.cgd.ucar.edu/cesm/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc
           => '/fs/cgd/csm/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc'
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.23.220
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.23.220|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /cesm/inputdata/lnd/clm2/initdata_map ... done.
==> SIZE clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc ... done.
==> PASV ... done.    ==> RETR clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc ...
No such file 'clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc'.

  Model clm missing file finidat = '/fs/cgd/csm/inputdata/lnd/clm2/initdata_map/clmi.I1850Clm45BgcGs.0901-01-01.0.9x1.25_gx1v7_simyr1850_c190718.nc'
ERROR from ftp server, trying next server
```

When I deleted the file, another test had the same problem.

This is with branch_tags/cime5.8.3_chint17-04 (similar to cime5.8.3).

Resolving patch:
diff --git a/scripts/lib/CIME/Servers/ftp.py b/scripts/lib/CIME/Servers/ftp.py
--- a/scripts/lib/CIME/Servers/ftp.py
+++ b/scripts/lib/CIME/Servers/ftp.py
@@ -7,6 +7,7 @@
 from ftplib import FTP as FTPpy
 from ftplib import all_errors as all_ftp_errors
 
+
 logger = logging.getLogger(__name__)
 # I think that multiple inheritence would be useful here, but I couldnt make it work
 # in a py2/3 compatible way.
@@ -50,6 +51,8 @@ def getfile(self, rel_path, full_path):
         try:
             stat = self.ftp.retrbinary('RETR {}'.format(rel_path), open(full_path, "wb").write)
         except all_ftp_errors:
+            if os.path.isfile(full_path):
+                os.remove(full_path)
             logger.warning("ERROR from ftp server, trying next server")
             return False
 
diff --git a/scripts/lib/CIME/case/check_input_data.py b/scripts/lib/CIME/case/check_input_data.py
--- a/scripts/lib/CIME/case/check_input_data.py
+++ b/scripts/lib/CIME/case/check_input_data.py
@@ -6,7 +6,7 @@
 from CIME.XML.inputdata import Inputdata
 import CIME.Servers
 
-import glob, hashlib
+import glob, hashlib, shutil
 
 logger = logging.getLogger(__name__)
 # The inputdata_checksum.dat file will be read into this hash if it's available
@@ -132,7 +132,10 @@ def _download_if_in_repo(server, input_data_root, rel_path, isdirectory=False):
             # this is intended to prevent a race condition in which
             # one case attempts to use a refdir before another one has
             # completed the download
-            os.rename(full_path+".tmp",full_path)
+            if success:
+                os.rename(full_path+".tmp",full_path)
+            else:
+                shutil.rmtree(full_path+".tmp")
         else:
             success = server.getfile(rel_path, full_path)
     return success


Test patch: