Repository: openstates/openstates-scrapers

Issue:
MN failing since at least 2018-07-28
MN has been failing since 2018-07-28

Based on automated runs it appears that MN has not run successfully in 2 days (2018-07-28).


```
  04:36:52 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SF4006&version=list
04:36:53 INFO pupa: save bill SF 4006 in 2017-2018 as bill_ec857344-9312-11e8-8091-02ddd4de1cca.json
04:36:53 INFO scrapelib: GET - https://www.revisor.mn.gov/bills/bill.php?b=Senate&f=SF4005&ssn=0&y=2018
04:36:54 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SF4005&version=list
04:36:55 INFO pupa: save bill SF 4005 in 2017-2018 as bill_eda9c0fe-9312-11e8-8091-02ddd4de1cca.json
04:36:55 INFO scrapelib: GET - https://www.revisor.mn.gov/bills/bill.php?b=Senate&f=SF4004&ssn=0&y=2018
04:36:56 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SF4004&version=list
04:36:57 INFO pupa: save bill SF 4004 in 2017-2018 as bill_eeda060a-9312-11e8-8091-02ddd4de1cca.json
04:36:57 INFO scrapelib: GET - https://www.revisor.mn.gov/bills/bill.php?b=Senate&f=SF4003&ssn=0&y=2018
04:36:58 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SF4003&version=list
04:36:59 INFO pupa: save bill SF 4003 in 2017-2018 as bill_f018d8ac-9312-11e8-8091-02ddd4de1cca.json
04:36:59 INFO scrapelib: GET - https://www.revisor.mn.gov/bills/bill.php?b=Senate&f=SF4002&ssn=0&y=2018
04:37:00 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SF4002&version=list
04:37:01 INFO pupa: save bill SF 4002 in 2017-2018 as bill_f1426fa4-9312-11e8-8091-02ddd4de1cca.json
04:37:01 INFO scrapelib: GET - https://www.revisor.mn.gov/bills/bill.php?b=Senate&f=SF4001&ssn=0&y=2018
04:37:02 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SF4001&version=list
04:37:03 INFO pupa: save bill SF 4001 in 2017-2018 as bill_f27ba908-9312-11e8-8091-02ddd4de1cca.json
04:37:03 INFO scrapelib: GET - https://www.revisor.mn.gov/bills/bill.php?b=Senate&f=SF4000&ssn=0&y=2018
04:37:04 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SF4000&version=list
04:37:05 INFO pupa: save bill SF 4000 in 2017-2018 as bill_f3a450c8-9312-11e8-8091-02ddd4de1cca.json
04:37:05 INFO scrapelib: GET - https://www.revisor.mn.gov/bills/bill.php?b=Senate&f=SC0011&ssn=0&y=2018
04:37:06 INFO scrapelib: GET - https://www.revisor.mn.gov/bin/getbill.php?session_year=2017&session_number=0&number=SC0011&version=list
loaded Open States pupa settings...
mn (scrape, import)
  bills: {}
  committees: {}
  people: {}
  vote_events: {}
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 113, in do_scrape
    for iterobj in obj:
  File "/opt/openstates/openstates/openstates/mn/bills.py", line 253, in get_bill_info
    bill = self.extract_versions(bill, doc, chamber, version_list_url)
  File "/opt/openstates/openstates/openstates/mn/bills.py", line 420, in extract_versions
    version_resp = self.get(version_list_url, verify=False)
  File "/opt/openstates/venv-pupa/lib/python3.6/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/opt/openstates/venv-pupa/lib/python3.6/site-packages/scrapelib/__init__.py", line 292, in request
    raise HTTPError(resp)
scrapelib.HTTPError: 404 while retrieving http://www.senate.mn/resolutions/display_resolution.php?ls=&bill_type=SC&bill_number=0011&ss_number=0&ss_year=2017
```

Visit http://bobsled.openstates.org for more info.



Discussion:
I'm working on this, looks like a bad URL on the state site.

Resolving patch:
diff --git a/openstates/mn/bills.py b/openstates/mn/bills.py
--- a/openstates/mn/bills.py
+++ b/openstates/mn/bills.py
@@ -1,6 +1,7 @@
 import re
 import datetime
 import urllib.parse
+import scrapelib
 from collections import defaultdict
 import lxml.html
 
@@ -417,7 +418,12 @@ def extract_versions(self, bill, doc, chamber, version_list_url):
         Versions of a bill are on a separate page, linked to from the column
         labeled, "Bill Text", on the search results page.
         """
-        version_resp = self.get(version_list_url, verify=False)
+        try:
+            version_resp = self.get(version_list_url, verify=False)
+        except scrapelib.HTTPError:
+            self.warning("Bad version URL detected: {}".format(version_list_url))
+            return bill
+
         version_html = version_resp.text
         if 'resolution' in version_resp.url:
             bill.add_version_link('resolution text', version_resp.url,


Test patch: