Repository: weecology/retriever

Issue:
Incorporate the PREDICTS database release
This is an impressive global database on species richness and abundance - it seems well suited for the `retriever`

http://data.nhm.ac.uk/dataset/902f084d-ce3f-429f-a6a5-23162c73fdf7


Discussion:
Totally agree.

I tried to do this right when it came out but there was no direct download link. I've been corresponding with the folks in charge and they provided this link:
http://data.nhm.ac.uk/dataset/the-2016-release-of-the-predicts-database/resource/78dac1a9-6aa0-4dcb-9750-50df04f8ca6e/download

We are allowed to use it for the retriever, but we're not supposed to advertise it more broadly.
ah yea I see now that they want your email address to download the files. Nice job getting that direct link!
@ethanwhite I would like to proceed on this. Shall I?
@jainamritanshu, go for it
@ethanwhite @henrykironde Sorry if I am missing something. I have created the json script of PREDICTS but the python script has not been created. I saw the comment in `def compile_json`(file `lib/compile.py`) that the command to do so is `retriever create_json <script name>`. When I do so, the cli tells me that it is an invalid argument. The exact error I got -
`retriever: error: argument command: invalid choice: 'create_json'`. Please help?
I think that the argument you are searching for here is ```new_json```. You can view all the options using ```retriever -h```.
@ShivamNegi I made the .json file using `new_json`. As I stated

> I have created the json script of PREDICTS but the python script has not been created.

But the .py file script wasn't created and as you can see in the screenshot -
![screenshot from 2017-03-03 21-22-35](https://cloud.githubusercontent.com/assets/15127027/23557865/931d4344-0057-11e7-98e7-1921bc68f471.png)

I saw the comment that stated that I need to execute the command `retriever create_json <script name>`. So I got confused. Thanks anyways. :)
Sorry, we forgot to change the docs. You could PR the changes to correct that. Nice find

@henrykironde Will do that :)

Resolving patch:
diff --git a/scripts/predicts.py b/scripts/predicts.py
new file mode 100644
--- /dev/null
+++ b/scripts/predicts.py
@@ -0,0 +1,124 @@
+#retriever
+
+"""Retriever script for direct download of data data"""
+from __future__ import print_function
+from builtins import str
+from future import standard_library
+
+standard_library.install_aliases()
+
+import os
+from retriever.lib.models import Table, Cleanup, correct_invalid_value
+from retriever.lib.templates import Script
+from retriever import VERSION
+from pkg_resources import parse_version
+
+
+class main(Script):
+    def __init__(self, **kwargs):
+        Script.__init__(self, **kwargs)
+        self.title = "PREDICTS Database"
+        self.name = "predicts"
+        self.ref = "http://data.nhm.ac.uk/dataset/902f084d-ce3f-429f-a6a5-23162c73fdf7"
+        self.urls = {"PREDICTS": "http://data.nhm.ac.uk/dataset/the-2016-release-of-the-predicts-database/"
+                                 "resource/78dac1a9-6aa0-4dcb-9750-50df04f8ca6e/download"}
+        self.citation = "Lawrence N Hudson; Tim Newbold; Sara Contu; Samantha L L Hill et al. (2016). Dataset: " \
+                        "The 2016 release of the PREDICTS database. http://dx.doi.org/10.5519/0066354"
+        self.keywords = ['biodiversity', 'anthropogenic pressures']
+        self.retriever_minimum_version = "2.0.dev"
+        self.version = "1.0.0"
+        self.description = "A dataset of 3,250,404 measurements, collated from 26,114 sampling locations in 94 " \
+                           "countries and representing 47,044 species."
+
+        if parse_version(VERSION) <= parse_version("2.0.0"):
+            self.shortname = self.name
+            self.name = self.title
+            self.tags = self.keywords
+
+    def download(self, engine=None, debug=False):
+        Script.download(self, engine, debug)
+        engine = self.engine
+
+        filename = "database.csv"
+        tablename = "predicts_main"
+
+        table = Table(str(tablename), delimiter=',')
+        table.columns = [("Source_ID", ("char",)),
+                         ("Reference", ("char",)),
+                         ("Study_number", ("int",)),
+                         ("Study_name", ("char",)),
+                         ("SS", ("char",)),
+                         ("Diversity_metric", ("char",)),
+                         ("Diversity_metric_unit", ("char",)),
+                         ("Diversity_metric_type", ("char",)),
+                         ("Diversity_metric_is_effort_sensitive", ("char",)),
+                         ("Diversity_metric_is_suitable_for_Chao", ("char",)),
+                         ("Sampling_method", ("char",)),
+                         ("Sampling_effort_unit", ("char",)),
+                         ("Study_common_taxon", ("char",)),
+                         ("Rank_of_study_common_taxon", ("char",)),
+                         ("Site_number", ("int",)),
+                         ("Site_name", ("char",)),
+                         ("Block", ("char",)),
+                         ("SSS", ("char",)),
+                         ("SSB", ("char",)),
+                         ("SSBS", ("char",)),
+                         ("Sample_start_earliest", ("char",)),
+                         ("Sample_end_latest", ("char",)),
+                         ("Sample_midpoint", ("char",)),
+                         ("Sample_date_resolution", ("char",)),
+                         ("Max_linear_extent_metres", ("double",)),
+                         ("Habitat_patch_area_square_metres", ("double",)),
+                         ("Sampling_effort", ("double",)),
+                         ("Rescaled_sampling_effort", ("double",)),
+                         ("Habitat_as_described", ("char",)),
+                         ("Predominant_land_use", ("char",)),
+                         ("Source_for_predominant_land_use", ("char",)),
+                         ("Use_intensity", ("char",)),
+                         ("Km_to_nearest_edge_of_habitat", ("double",)),
+                         ("Years_since_fragmentation_or_conversion", ("double",)),
+                         ("Transect_details", ("char",)),
+                         ("Coordinates_method", ("char",)),
+                         ("Longitude", ("double",)),
+                         ("Latitude", ("double",)),
+                         ("Country_distance_metres", ("double",)),
+                         ("Country", ("char")),
+                         ("UN_subregion", ("char",)),
+                         ("UN_region", ("char",)),
+                         ("Ecoregion_distance_metres", ("double",)),
+                         ("Ecoregion", ("char",)),
+                         ("Biome", ("char",)),
+                         ("Realm", ("char",)),
+                         ("Hotspot", ("char",)),
+                         ("Wilderness_area", ("char",)),
+                         ("N_samples", ("double",)),
+                         ("Taxon_number", ("double",)),
+                         ("Taxon_name_entered", ("char",)),
+                         ("Indication", ("char",)),
+                         ("Parsed_name", ("char",)),
+                         ("Taxon", ("char",)),
+                         ("COL_ID", ("double",)),
+                         ("Name_status", ("char",)),
+                         ("Rank", ("char",)),
+                         ("Kingdom", ("char",)),
+                         ("Phylum", ("char",)),
+                         ("Class", ("char",)),
+                         ("Order", ("char",)),
+                         ("Family", ("char",)),
+                         ("Genus", ("char",)),
+                         ("Species", ("char",)),
+                         ("Best_guess_binomial", ("char",)),
+                         ("Higher_taxa", ("char",)),
+                         ("Higher_taxon", ("char",)),
+                         ("Measurement", ("double",)),
+                         ("Effort_corrected_measurement", ("double",))]
+
+        engine.table = table
+        if not os.path.isfile(engine.format_filename(filename)):
+            engine.download_files_from_archive(self.urls["PREDICTS"], [filename], filetype="zip",
+                                               archivename="download.zip")
+        engine.create_table()
+        engine.insert_data_from_file(engine.format_filename(str(filename)))
+
+
+SCRIPT = main()


Test patch: