Package: retriever
Finding data is one thing. Getting it ready for analysis is another. Acquiring,
cleaning, standardizing and importing publicly available data is time consuming
because many datasets lack machine readable metadata and do not conform to
established data structures and formats. The Data Retriever automates the first
steps in the data analysis pipeline by downloading, cleaning, and standardizing
datasets, and importing them into relational databases, flat files, or
programming languages. The automation of this process reduces the time for a
user to get most large datasets up and running by hours, and in some cases days.Installing the Current ReleaseIf you have Python installed you can install the current release using eitherpip:pipinstallretrieverorcondaafter adding theconda-forgechannel (conda config --add channels conda-forge):condainstallretrieverDepending on your system configuration this may requiresudoforpip:sudopipinstallretrieverPrecompiled binary installers are also available for Windows, OS X, and
Ubuntu/Debian on
thereleases page. These do
not require a Python installation.List of Available DatasetsInstalling From SourceTo install the Data Retriever from source, you'll need Python 3.6.8+ with the following packages installed:xlrdThe following packages are optionally needed to interact with associated
database management systems:PyMySQL (for MySQL)sqlite3 (for SQLite)psycopg2-binary (for PostgreSQL), previously psycopg2.pyodbc (for MS Access - this option is only available on Windows)Microsoft Access Driver (ODBC for windows)To install from sourceEither usepipto install directly from GitHub:pipinstallgit+https://git@github.com/weecology/retriever.gitor:Clone the repositoryFrom the directory containing setup.py, run the following command:pip install .. You may need to includesudoat the beginning of the
command depending on your system (i.e.,sudo pip install .).More extensive documentation for those that are interested in developing can be foundhereUsing the Command LineAfter installing, runretriever updateto download all of the available dataset scripts.
To see the full list of command line options and datasets runretriever --help.
The output will look like this:usage:retriever[-h][-v][-q]{download,install,defaults,update,new,new_json,edit_json,delete_json,ls,citation,reset,help}...

positionalarguments:{download,install,defaults,update,new,new_json,edit_json,delete_json,ls,citation,reset,help}sub-commandhelpdownloaddownloadrawdatafilesforadatasetinstalldownloadandinstalldatasetdefaultsdisplaysdefaultoptionsupdatedownloadupdatedversionsofscriptsnewcreateanewsampleretrieverscriptnew_jsonCLItocreateretrieverdatapackage.jsonscriptedit_jsonCLItoeditretrieverdatapackage.jsonscriptdelete_jsonCLItoremoveretrieverdatapackage.jsonscriptlsdisplayalistallavailabledatasetscriptscitationviewcitationresetresetretriever:removesconfigurationsettings,scripts,andcacheddatahelpoptionalarguments:-h,--helpshowthishelpmessageandexit-v,--versionshowprogram'sversionnumberandexit-q,--quietsuppresscommand-lineoutputTo install datasets, useretriever install:usage:retrieverinstall[-h][--compile][--debug]{mysql,postgres,sqlite,msaccess,csv,json,xml}...

positionalarguments:{mysql,postgres,sqlite,msaccess,csv,json,xml}engine-specifichelpmysqlMySQLpostgresPostgreSQLsqliteSQLitemsaccessMicrosoftAccesscsvCSVjsonJSONxmlXML

optionalarguments:-h,--helpshowthishelpmessageandexit--compileforcere-compileofscriptbeforedownloading--debugrunindebugmodeExamplesThese examples are using theIrisflower dataset.
More examples can be found in the Data Retriever documentation.Using Installretrieverinstall-h(givesinstalloptions)Using specific database engine, retriever install {Engine}retrieverinstallmysql-h(givesinstallmysqloptions)retrieverinstallmysql--usermyuser--password********--hostlocalhost--port8888--database_nametestdbaseirisinstall data into an sqlite database named iris.db you would use:retrieverinstallsqliteiris-firis.dbUsing downloadretrieverdownload-h(givesyouhelpoptions)retrieverdownloadiris
retrieverdownloadiris--pathC:\Users\DocumentsUsing citationretrievercitation(citationoftheretrieverengine)retrievercitationiris(citationfortheirisdata)Spatial Dataset InstallationSet up Spatial supportTo set up spatial support for Postgres using Postgis please
refer to thespatial set-up docs.retrieverinstallpostgresharvard-forest# Vector dataretrieverinstallpostgresbioclim# Raster data# Install only the data of USGS elevation in the given extentretrieverinstallpostgresusgs-elevation-b-94.9870459735393839.027001800158615-94.359940811991740.69577051867074WebsiteFor more information see theData Retriever website.AcknowledgmentsDevelopment of this software was funded by theGordon and Betty Moore
Foundation's Data-Driven Discovery
InitiativethroughGrant GBMF4563to Ethan White and
theNational Science Foundationas part of aCAREER award to
Ethan White.