vtsuperdarn/davitpy — pull request #108 by asreimer (2014-09-29 07:09:26+00:00)
Title: Fixed ctime bug

**The Bug**
The ctime bug is caused by databases where files are not stored with a common time increment between them. Put another way, fetchUtils was operating under the false assumption that files were stored with a common time increment between them. Relevant bug: https://github.com/vtsuperdarn/davitpy/issues/24

**The Fix**
The `time_inc` variable has been removed from both `fetch_local_files` and `fetch_remote_files` since it is not needed (since we can't assume files are separated by a common time increment). fetchUtils now checks to see which time fields were specified in `fnamefmt`. The "smallest" time field is then used to increment/decrement `ctime`. First, `ctime` is set to user specified `stime`. Second, `ctime` is decremented until a file is found (this ensures that we have data coverage up to `stime`. Third, `ctime` is incremented until it hits the user specified `etime`.

TL;DR `ctime` is now decremented/incremented. Increment set by smallest time field in `fnamefmt`.

**Extra**
A new time field `min` is now supported by fetchUtils. Added a `check_cache` variable to force fetchUtils to skip checking for cached files.

**Testing**
To test this pull request, simply execute `python fetchUtils.py` or from within DaViTpy run the following:
`pydarn.sdio.fetchUtils.fetch_remote_files(datetime(2012,11,24,3,30), datetime(2012,11,24,7),'sftp','sd-data.ece.vt.edu','data/{year}/{ftype}/{radar}/',{'ftype':"fitex",'radar':"kod",'channel':'.'}, '/tmp/sd/',['{date}.{hour}......{radar}.{ftype}','{date}.{hour}......{radar}.{channel}.{ftype}'], username='sd_dbread', password='5d')`

To test forcing file download run this:
`pydarn.sdio.fetchUtils.fetch_remote_files(datetime(2012,11,24,4,30), datetime(2012,11,24,7),'sftp','sd-data2.ece.vt.edu','data/{year}/{ftype}/{radar}/',{'ftype':"fitex",'radar':"kod",'channel':'.'}, '/tmp/sd/',['{date}.{hour}......{radar}.{ftype}','{date}.{hour}......{radar}.{channel}.{ftype}'], username='sd_dbread', password='5d',verbose=True,check_cache=True)`
and files should download.

then run this:
`pydarn.sdio.fetchUtils.fetch_remote_files(datetime(2012,11,24,4,30), datetime(2012,11,24,7),'sftp','sd-data2.ece.vt.edu','data/{year}/{ftype}/{radar}/',{'ftype':"fitex",'radar':"kod",'channel':'.'}, '/tmp/sd/',['{date}.{hour}......{radar}.{ftype}','{date}.{hour}......{radar}.{channel}.{ftype}'], username='sd_dbread', password='5d',verbose=True,check_cache=False)`
and files should still download even though they are cached.