Repository: openstates/openstates-scrapers

Issue:
CO scraper failing since at least 2017-05-04
CO has been failing since 2017-05-04

Based on automated runs it appears that CO has not run successfully in 2 days (2017-05-04).


```
  07:06:46 INFO billy: Save vote 2017A lower: SB17-184 'Report. As shown by the following roll call vote, a majority of those elected to the House voted in the affirmative, and the Report was adopted.'
07:06:46 INFO scrapelib: GET - http://www.leg.state.co.us/CLICS/CLICS2017A/csljournals.nsf/(jouhse)/16B4F6F45CAAB946872580F200514BE8/$FILE/Ma29.pdf
07:06:47 INFO billy: Save vote 2017A lower: HB17-1219 'The question being "Shall the bill pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative and the bill was declared passed.'
07:06:47 INFO billy: Save vote 2017A lower: HB17-1236 'The question being "Shall the bill pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative and the bill was declared passed.'
07:06:47 INFO billy: Save vote 2017A lower: HB17-1256 'The question being "Shall the bill pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative and the bill was declared passed.'
07:06:47 INFO billy: Save vote 2017A lower: SB17-112 'The question being "Shall the bill pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative and the bill was declared passed.'
07:06:47 INFO billy: Save vote 2017A lower: HB17-1035 'Representative Jackson moved that the House not concur in Senate amendments and that a Conference Committee be appointed. The motion was declared passed by the following roll call vote:'
07:06:47 INFO scrapelib: GET - http://www.leg.state.co.us/CLICS/CLICS2017A/csljournals.nsf/(jouhse)/800C0A6E638164DB872580F3004F26F8/$FILE/Ma30.pdf
07:06:48 INFO billy: Save vote 2017A lower: HB17-1230 'The amendment was declared passed by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1230 'The question being, "Shall the bill, as amended, pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative, and the bill, as amended, was declared passed.'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared lost by the following roll call vote:'
07:06:48 INFO billy: Save vote 2017A lower: HB17-1242 'The amendment was declared passed by the following roll call vote:'
07:06:48 INFO scrapelib: GET - http://www.leg.state.co.us/CLICS/CLICS2017A/csljournals.nsf/(jouhse)/FA7A7FA24DB49B1C872580F400520EBC/$FILE/Ma31.pdf
07:06:49 INFO billy: Save vote 2017A lower: SB17-230 'The question being "Shall the bill pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative and the bill was declared passed.'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1242 'The question being "Shall the bill pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative and the bill was declared passed.'
07:06:49 INFO billy: Save vote 2017A lower: SB17-213 'Report. As shown by the following roll call vote, a majority of those elected to the House voted in the affirmative, and the Report was adopted.'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1049 'Representative Gray moved that the House concur in Senate amendments. The motion was declared passed by the following roll call vote:'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1049 'The question being, "Shall the bill, as amended, pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative, and the bill, as amended, was declared repassed.'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1174 'Representative Wilson moved that the House concur in Senate amendments. The motion was declared passed by the following roll call vote:'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1174 'The question being, "Shall the bill, as amended, pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative, and the bill, as amended, was declared repassed.'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1193 'Representative Kraft-Tharp moved that the House concur in Senate amendments. The motion was declared passed by the following roll call vote:'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1193 'The question being, "Shall the bill, as amended, pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative, and the bill, as amended, was declared repassed.'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1220 'Representative KC Becker moved that the House concur in Senate amendments. The motion was declared passed by the following roll call vote:'
07:06:49 INFO billy: Save vote 2017A lower: HB17-1220 'The question being, "Shall the bill, as amended, pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative, and the bill, as amended, was declared repassed.'
07:06:49 INFO scrapelib: GET - http://www.leg.state.co.us/CLICS/CLICS2017A/csljournals.nsf/(jouhse)/084B4DAFD7AEEF858725811400527F98/$FILE/My02.pdf
07:06:50 INFO billy: Save vote 2017A lower: HB17-1324 'The question being "Shall the bill pass?". A roll call vote was taken. As shown by the following recorded vote, a majority of those elected to the House voted in the affirmative and the bill was declared passed.'
07:06:50 WARNING billy: Value None for field '<obj>.date' is not of type datetime
  File "/srv/openstates-web/openstates/co/votes.py", line 338, in scrape
  File "/usr/local/bin/billy-update", line 9, in <module>
    load_entry_point('billy==1.9.0', 'console_scripts', 'billy-update')()
  File "/opt/openstates/billy/billy/bin/update.py", line 368, in main
    run_record += _run_scraper(stype, args, metadata)
  File "/opt/openstates/billy/billy/bin/update.py", line 102, in _run_scraper
    scraper.scrape(chamber, time)
Traceback (most recent call last):
    self.scrape_house(session)
  File "/srv/openstates-web/openstates/co/votes.py", line 175, in scrape_house
    self.save_vote(vote)
  File "/opt/openstates/billy/billy/scrape/__init__.py", line 199, in save_object
    self.validate_json(obj)
  File "/opt/openstates/billy/billy/scrape/__init__.py", line 130, in validate_json
    raise ve
validictory.validator.FieldValidationError: Value None for field '<obj>.date' is not of type datetime
```

Visit http://bobsled.openstates.org/ for more info.



Discussion:

Resolving patch:
diff --git a/openstates/co/votes.py b/openstates/co/votes.py
--- a/openstates/co/votes.py
+++ b/openstates/co/votes.py
@@ -21,6 +21,11 @@
            "ABSENT\s*(?P<abs_count>\d+).*"))
 votes_re = r"(?P<name>\w+(\s\w\.)?)\s+(?P<vote>Y|N|A|E|-)"
 
+
+def fix_typos(data):
+    return data.replace('Tueday', 'Tuesday')  # Spelling is hard
+
+
 class COVoteScraper(VoteScraper, LXMLMixin):
     jurisdiction = 'co'
 
@@ -32,6 +37,7 @@ def scrape_house(self, session):
         for href in hrefs:
             (path, response) = self.urlretrieve(href.attrib['href'])
             data = convert_pdf(path, type='text')
+            data = fix_typos(data)
 
             in_vote = False
             cur_vote = {}
@@ -202,6 +208,7 @@ def scrape_senate(self, session):
         for href in hrefs:
             (path, response) = self.urlretrieve(href.attrib['href'])
             data = convert_pdf(path, type='text')
+            data = fix_typos(data)
 
             cur_bill_id = None
             cur_vote_count = None


Test patch: