Package: ecohydrolib
EcohydroLib			{#index}=============================This software is provided free of charge under the New BSD License. Please seethe following license information:Copyright (c) 2013-2015 University of North Carolina at Chapel HillAll rights reserved.Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditions are met:- Redistributions of source code must retain the above copyrightnotice, this list of conditions and the following disclaimer.- Redistributions in binary form must reproduce the above copyrightnotice, this list of conditions and the following disclaimer in thedocumentation and/or other materials provided with the distribution.- Neither the name of the University of North Carolina at Chapel Hill northe names of its contributors may be used to endorse or promote productsderived from this software without specific prior written permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" ANDANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AREDISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILLBE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, ORCONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTEGOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICTLIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUTOF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.Authors-------Brian Miles <brian_miles@unc.edu>Lawrence E. Band <lband@email.unc.edu>For questions or support contact [Brian Miles](brian_miles@unc.edu)Funding-------This work was supported by the following NSF grants- Award no. 1239678 EAGER: Collaborative Research: InteroperabilityTestbed-Assessing a Layered Architecture for Integration ofExisting Capabilities- Award no. 0940841 DataNet Federation Consortium.- Award no. 1148090 Collaborative Research: SI2-SSI: An Interactive SoftwareInfrastructure for Sustaining Collaborative Innovation in theHydrologic SciencesIntroduction------------EcohydroLib provides a series of Python scripts for performingecohydrology data preparation workflows.  Workflow sub-components areorchestrated via a metadata persistence store provided by theecohydrolib.metadata package.  These scripts are built on top of aseries of task-oriented APIs defined in the Python packageEcohydroLib.  The workflow scripts provide tools for downloading andmanipulating geospatial data needed to run a ecohydrology models,information such as: digital elevation model (DEM), soils, land cover,and vegetation leaf area index.  These data can be drawn both fromnational spatial data infrastructure (NLCD, SSURGO) as well as customlocal datasets.A metadata store is used to orchestrate a series of workflow scriptsused to prepare data for an ecohydrology model.  The currentimplementation uses the Python ConfigParser key-value storagemechanism to persist metadata to disk, however conceivably anykey-value store could be used.  The metadata contains informationrelated to the study area (e.g. bounding box, spatial reference, DEMresolution), as well as provenance information for each spatial datalayer imported, and a processing history that records the parametersused to invoke each workflow script; provenance information isrepresented as a subset of Dublin Core attributes(http://dublincore.org).  When using the workflow scripts in astand-alone environment, the metadata store provides informationnecessary to understand where ecohydrology input data came from, andwhat transformations have been made to the data.  When the workflowscripts are integrated into a data grid or workflow environment(e.g. iRODS), the metadata store can serve as a staging area formetadata and provenance information that will be registered into theformal workflow environment.The fundamental operation for any ecohydrology modeling workflow is todefine the study region of interest (ROI).  In EcohydroLib the ROI issimply defined as a bounding box of WGS84 latitude and longitudecoordinates (e.g. coordinates for the upper- left and lower-rightcorners).  For workflows using the National Hydrography Dataset (NHD),the ROI bounding box can be derived using catchment polygonsassociated with the stream reaches upstream of a particular gage.  Theuser begins by picking a streamflow discharge gage listed in the NHDdataset.  EcohydroLib can then determine the stream reaches upstreamof the data, and then select the catchment polygons associated witheach upstream reach.  From these polygons, the bounding box of theland area draining through the streamflow gage can easily becalculated.Once the ROI is known, EcohydroLib can extract datasets (DEM, soils,etc.)  specific to the study area.  Some of these datasets areextracted from static local copies of national spatial data(e.g. NLCD), while other are retrieved via web services interfacesfrom federal agency data centers (e.g. SSURGO soils data from USDA) orfrom third-party data centers (GeoBrain's DEM Explorer).  However itis also possible for the user to register their own custom data for agiven datatype (e.g. local LIDAR-based DEM).![Fig. 1 Ecohydrology model data preparation workflow software stack depicting EcohydroLib's role as an intermediary between raw data, derived data subsets, and specific ecohydrology models](EcohydroLib-Architecture.png)Source code-----------Source code can be found at: https://github.com/selimnairb/EcohydroLib.Documentation can be found at: http://pythonhosted.org/ecohydrolibInstallation------------Detailed installation and usage instructions can be found in the RHESSysWorkflows[readme](https://github.com/selimnairb/RHESSysWorkflows).The following instructions should only be used by advanced users.Using easy_install:easy_install --script-dir /path/to/install/scripts EcohydroLibUsing pip:pip install EcohydroLibIt is recommended that you install the workflow scripts in a locationdistinct from where the Python package will be installed.  This isaccomplished by specifying the --script-dir option to easy install(see above).Note, pyspatialite 3.0.1, needed for GHCNDSetup.py andGetGHCNDailyClimateData*.py, currently fails to build under easy_install/pip.Until this is fixed by the pyspatialite developer, I have removedpyspatialite from the dependency list.  If you need to use GNCHD data, youcan install pyspatialite manually using the following steps (this can bedone before or after installing EcohydroLib):- Manually download pyspatialite here:https://pypi.python.org/pypi/pyspatialite/3.0.1- Apply the following patch to pyspatialite's setup.py:https://code.google.com/p/pyspatialite/issues/detail?id=9- Install pyspatialite- Install EcohydroLib as described aboveRequired runtime software-------------------------Python 2.7Libraries (with headers):- libxml2- libxslt- libproj- libgeosBinaries:- GDAL/OGR 1.9 or later (throughout)- SQLite3 (throughout)- Seven Zip (if using NHDPlusV2Setup/NHDPlusV2Setup.py)- Spatialite (if using GHCNDSetup.py/GetGHCNDailyClimateData*.py)- Unix find (if using NHDPlusV2Setup/NHDPlusV2Setup.py)Data stored locally-------------------- NLCD 2006 raster (http://www.mrlc.gov/nlcd06_data.php)- HYDRO1k North America dataset (http://eros.usgs.gov/#/Find_Data/Products_and_Data_Available/gtopo30/hydro/namerica)- NHDPlus V2 dataset (http://www.horizon-systems.com/NHDPlus/NHDPlusV2_home.php)NHDPlus V2 database setup-------------------------Before EcohydroLib is able to extract study area ROI using the NHDPlusdataset, it is necessary to have a local copy of the NHDPlus dataset.Owing to the large size of the NHDPlus dataset, these data aredistributed as as series of compressed archives broken into severalregions for the continental U.S.  There are two choices for obtainingNHDPlus in a format usable by EcohydroLib (as several SQLite3databases).  A national-scale dataset (i.e. covering the entirecontinental U.S.) is available for download here:http://...Once downloaded, extract the archive and record its location in yourEcohydroLib configuation file; see the 'Configuration files' sectionfor more information.If you wish to build you own copy of the database (i.e. for a subsetof U.S. country) a script for building the dataset from downloadedNHDPlus V2 7z archives is provided inbin/NHDPlusV2Setup/NHDPlusV2Setup.py.  The following NHDPlus V2datasets are required:- NHDPlusV21_NationalData_GageInfo_02.7z- NHDPlusV21_NationalData_GageLoc_01.7z- NHDPlusV21_NationalData_Gage_Smooth_01.7z- NHDPlusV21_??_??_NHDPlusAttributes_??.7z- NHDPlusV21_??_??_NHDPlusCatchment_??.7z- NHDPlusV21_??_??_NHDSnapshot_??.7zNote that the NHDPlusAttributes, NHDPlusCatchment, and NHDPlusSnapshotdata are released as regional subsets (due to the large size andcomplexity of the data).  NHDPlusV2Setup.py can build its NHDPlusSQLite3 databases for any number of regions; all data for the desirednumber of regions will be combined into a single database.Once you've decided which NHDPlusV2 regions you wish to build adatabase for, simply download the relevant 7z archives from theNHDPlusV2 web site (see above), and store the archives in a singledirectory.  NHDPlusV2Setup.py will unpack these archives into aspecified output location and then will process the unarchived filesinto the following databases: - Catchment.sqlite (a spatial datasetcontaining all catchment polygons in the selected NHD region(s); -GageLoc.sqlite (a spatial dataset containing streamflow gage pointsfor the national NHD dataset; - NHDPlusDB.sqlite (a tabular datasetcontaining other NHD data needed by EcohydroLib).Make sure to edit your configuration file to include the absolutepaths of these files (see below).For national NHD coverage, Catchment.sqlite is over 8 GB, andNHDPlusDB.sqlite is over 2 GB, so you will need a kernel andfilesystem that has large file support to build and use thesedatasets.  Also, it may take over an hour to create these datasets; 8GB of memory or more is recommended to build the datasets efficiently.However, database setup is a one-time process, and you can usedatabases created on one machine on other machines, provided SQLite3is installed.  NHDPlusV2Setup.py creates each database with theindices needed by EcohydroLib, so lookups are very fast.HYDRO1k North America---------------------To use HYDRO1k basin shapefile, you must first uncompressna_bas.e00.gz to na_bas.e00.  Then you must convert the e00 (Arcinterchange file) to a shapefile using a tool such as ArcGIS.GHCN Climate Data-----------------To download NCDC Global Historical Climatology Network (GHCN) datasetfor daily climate data, you must first create the spatialite databasethat EcohydroLib uses to find climate stations usingspatial queries.  This database is created usingbin/GHCNDSetup/GHCNDSetup.py.  The output from the script will be aspatialite database.  Make sure to edit your configuration file andset PATH_OF_STATION_DB to the absolute path of this spatialitedatabase (see below).Configuration files-------------------Many of the command line scripts (including NHDPlusV2Setup.py) requirea configuration file to specify locations to executables and datasetsrequired by the ecohydrology workflow libraries.  The configurationfile can be specified via the environmental variableECOHYDROLIB_CFG or via command line option. Here is an exampleconfiguration file:[GDAL/OGR]PATH_OF_OGR2OGR = /Library/Frameworks/GDAL.framework/Versions/Current/Programs/ogr2ogrPATH_OF_GDAL_RASTERIZE = /Library/Frameworks/GDAL.framework/Versions/Current/Programs/gdal_rasterizePATH_OF_GDAL_WARP = /Library/Frameworks/GDAL.framework/Versions/Current/Programs/gdalwarpPATH_OF_GDAL_TRANSLATE = /Library/Frameworks/GDAL.framework/Versions/Current/Programs/gdal_translate[NHDPLUS2]PATH_OF_NHDPLUS2_DB = /Users/<username>/Research/data/GIS/NHDPlusV21/national/NHDPlusDB.sqlitePATH_OF_NHDPLUS2_CATCHMENT = /Users/<username>/Research/data/GIS/NHDPlusV21/national/Catchment.sqlitePATH_OF_NHDPLUS2_GAGELOC = /Users/<username>/Research/data/GIS/NHDPlusV21/national/GageLoc.sqlite[SOLIM]PATH_OF_SOLIM = /Users/<username>/Research/bin/solim/solim.out[NLCD]PATH_OF_NLCD2006 = /Users/<username>/Research/data/GIS/NLCD2006/nlcd2006/nlcd2006_landcover_4-20-11_se5.img[HYDRO1k]PATH_OF_HYDRO1K_DEM = /Users/<username>/Research/data/GIS/HYDRO1k/na/na_dem.bilPATH_OF_HYDRO1K_BAS = /Users/<username>/Research/data/GIS/HYDRO1k/na/na_bas_polygon.shpHYDRO1k_BAS_LAYER_NAME = na_bas_polygon[GHCND]PATH_OF_STATION_DB = /Users/<username>/Research/data/obs/NCDC/GHCND/GHCND.spatialite[UTIL]PATH_OF_FIND = /usr/bin/findPATH_OF_SEVEN_ZIP = /opt/local/bin/7zPATH_OF_SQLITE = /opt/local/bin/sqlite3If you create your initial configuration file by copying and pastingfrom this documentation, make sure to remove any leading spaces fromeach line of the file.How to use - Typical workflows------------------------------A workflow using data from large-scale spatial data infrastructurewill consist of running the follow scripts in the following order:1. GetNHDStreamflowGageIdentifiersAndLocation.py2. GetCatchmentShapefileForNHDStreamflowGage.py3. GetBoundingboxFromStudyAreaShapefile.py4. GetUSGSDEMForBoundingbox.py5. GetNLCDForDEMExtent.py6. GetSSURGOFeaturesForBoundingbox.py7. GenerateSoilPropertyRastersFromSSURGO.py or GenerateSoilPropertyRastersFromSOLIM.pyThe first 4 steps must be run in this order, the remaining workflowcomponents can be run in any order.  Other workflow components,e.g. to register a custom dataset, can be substituted for the latter 4workflow components as well (as indicated above).  See thedocumentation for each script to see invocations details.A workflow collecting data appropriate for large-scale land surfaceprocess models may consist of running the following scripts in thefollowing order:1. GetCatchmentShapefileForHYDRO1kBasins.py2. GetBoundingboxFromStudyAreaShapefile.py3. GetHYDRO1kDEMForBoundingbox.py6. GetGHCNDailyClimateDataForBoundingboxCentroid.py OR GetGHCNDailyClimateDataForStationsInBoundingbox.pyA workflow using custom local data sources will consist of running thefollow scripts in the following order:1. RegisterDEM.py2. RegisterGage.py3. RegisterRaster.py4. GetSSURGOFeaturesForBoundingbox.pyA workflow using custom streamflow gage, but with standard spatial data (NED, NLCD, SSURGO)could consist of running the follow scripts in the following order:1. RegisterStudyAreaShapefile.py2. GetBoundingboxFromStudyAreaShapefile.py3. GetUSGSDEMForBoundingbox.py4. GetNLCDForDEMExtent.py5. GetSSURGOFeaturesForBoundingbox.py6. GenerateSoilPropertyRastersFromSSURGO.py