Repository: weecology/retriever

Issue:
breed-bird-survey broken due to change by data source
Looks like BBS did it's annual update and we're broken again:

```
(base) ethan@oneesk:~$ retriever install sqlite breed-bird-survey
=> Installing breed-bird-survey

Saving a copy of SpeciesList.txt...
Downloading SpeciesList.txt: 280kB [00:01, 159kB/s]                                                                                                                                                                
'utf-8' codec can't decode byte 0xe0 in position 1565: invalid continuation byte
Done!
```


Discussion:
Ethan, I was using the rdataretriever dataset to access the bbs (counts dataset specifically, 2001-2015) and the rpid column seems to have been added in, duplicating some routes in my dataset. Now I have to have a line filtering out certain rpid codes to remove duplicates.  I was using the same code before this with no duplicates, is it possible these settings also changed with this update?
Hi Sara, sorry for the delay on this. I am getting it fixed right away. Since the folks from bbs change the data annually, It may be hard to get your analysis looking exactly the same. 

`The good news is that we have added a new feature, Provenance, to the current master that enables you to commit and archive datasets and installation of the committed dataset at a later date. This ensures that the previous outputs/results can be produced easily` 
**Provenance:** https://retriever.readthedocs.io/en/latest/provenance.html


thanks so much, this is really helpful!
@ssnell6 I think I got breed-bird-survey(BBS) to work locally, I will put in the changes in the evening.
The new version of BBS has some changes with the encoding which we will need to handle and a few file changes that are not mentioned in the `whatsnew.txt`
After which, we shall be good to run BBS.

ref: ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/
@ssnell6, What operating system are you using?
windows

Resolving patch:
diff --git a/scripts/breed_bird_survey.py b/scripts/breed_bird_survey.py
--- a/scripts/breed_bird_survey.py
+++ b/scripts/breed_bird_survey.py
@@ -44,7 +44,7 @@ def __init__(self, **kwargs):
         self.ref = "http://www.pwrc.usgs.gov/BBS/"
         self.keywords = ["birds", "continental-scale"]
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '2.0.0'
+        self.version = '3.0.0'
         self.urls = {
             "counts": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/States/",
             "routes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/routes.zip",
@@ -61,6 +61,7 @@ def __init__(self, **kwargs):
             self.cleanup_func_clean = Cleanup(
                 correct_invalid_value, nulls=['*'])
         else:
+            self.encoding = "latin-1"
             self.cleanup_func_table = Cleanup(
                 correct_invalid_value, missing_values=['NULL'])
             self.cleanup_func_clean = Cleanup(
@@ -74,7 +75,7 @@ def download(self, engine=None, debug=False):
 
             # Species table
             table = Table("species", cleanup=Cleanup(), contains_pk=True,
-                          header_rows=9)
+                          header_rows=10)
 
             table.columns = [("species_id", ("pk-int",)),
                              ("AOU", ("int",)),
@@ -128,7 +129,7 @@ def download(self, engine=None, debug=False):
                 engine.format_filename("weather_new.csv"))
 
             # Region_codes table
-            table = Table("region_codes", pk=False, header_rows=11,
+            table = Table("region_codes", pk=False, header_rows=12,
                           fixed_width=[11, 11, 30])
 
             def regioncodes_cleanup(value, engine):
diff --git a/scripts/breed_bird_survey_50stop.py b/scripts/breed_bird_survey_50stop.py
--- a/scripts/breed_bird_survey_50stop.py
+++ b/scripts/breed_bird_survey_50stop.py
@@ -38,7 +38,7 @@ def __init__(self, **kwargs):
         self.ref = "http://www.pwrc.usgs.gov/BBS/"
         self.keywords = ["birds", "continental-scale"]
         self.retriever_minimum_version = '2.0.dev'
-        self.version = '1.4.4'
+        self.version = '2.0.0'
         self.urls = {
             "counts": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/50-StopData/1997ToPresent_SurveyWide/",
             "routes": "ftp://ftpext.usgs.gov/pub/er/md/laurel/BBS/DataFiles/routes.zip",
@@ -55,6 +55,7 @@ def __init__(self, **kwargs):
             self.cleanup_func_clean = Cleanup(
                 correct_invalid_value, nulls=['*'])
         else:
+            self.encoding = "latin-1"
             self.cleanup_func_table = Cleanup(
                 correct_invalid_value, missing_values=['NULL'])
             self.cleanup_func_clean = Cleanup(
@@ -68,7 +69,7 @@ def download(self, engine=None, debug=False):
 
             # Species table
             table = Table("species", cleanup=Cleanup(), contains_pk=True,
-                          header_rows=9)
+                          header_rows=10)
 
             table.columns = [("species_id", ("pk-int",)),
                              ("AOU", ("int",)),
@@ -121,7 +122,7 @@ def download(self, engine=None, debug=False):
                 engine.format_filename("weather_new.csv"))
 
             # Region_codes table
-            table = Table("region_codes", pk=False, header_rows=11,
+            table = Table("region_codes", pk=False, header_rows=12,
                           fixed_width=[11, 11, 30])
 
             def regioncodes_cleanup(value, engine):


Test patch: