Repository: openstates/openstates-scrapers

Issue:
KY failing since at least 2019-03-17
KY has been failing since 2019-03-17

Based on automated runs it appears that KY has not run successfully in 2 days (2019-03-17).


```
  06:08:01 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr178.html
06:08:02 INFO pupa: save bill sr178 in 2019RS as bill_3045ed06-4944-11e9-a67a-02561413082c.json
06:08:02 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr179.html
06:08:03 INFO pupa: save bill sr179 in 2019RS as bill_30df30ba-4944-11e9-a67a-02561413082c.json
06:08:03 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr180.html
06:08:04 INFO pupa: save bill sr180 in 2019RS as bill_3177c758-4944-11e9-a67a-02561413082c.json
06:08:04 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr181.html
06:08:05 INFO pupa: save bill sr181 in 2019RS as bill_32105e1e-4944-11e9-a67a-02561413082c.json
06:08:05 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr182.html
06:08:06 INFO pupa: save bill sr182 in 2019RS as bill_32ac3dfc-4944-11e9-a67a-02561413082c.json
06:08:06 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr183.html
06:08:07 INFO pupa: save bill sr183 in 2019RS as bill_33418772-4944-11e9-a67a-02561413082c.json
06:08:07 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr184.html
06:08:08 INFO pupa: save bill sr184 in 2019RS as bill_33da4656-4944-11e9-a67a-02561413082c.json
06:08:08 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr185.html
06:08:09 INFO pupa: save bill sr185 in 2019RS as bill_3472fe00-4944-11e9-a67a-02561413082c.json
06:08:09 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr186.html
06:08:10 INFO pupa: save bill sr186 in 2019RS as bill_350bef66-4944-11e9-a67a-02561413082c.json
06:08:10 INFO scrapelib: GET - https://apps.legislature.ky.gov/record/19RS/sr188.html
loaded Open States pupa settings...
ky (scrape, import)
  bills: {}
Traceback (most recent call last):
  File "/opt/**PGUSER**/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 278, in handle
    return self.do_handle(args, other, juris)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 327, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 175, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/scrape/base.py", line 112, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ky/bills.py", line 62, in scrape
    yield from self.scrape_session(chamber, session)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ky/bills.py", line 72, in scrape_session
    yield from self.scrape_bill_list(chamber, session, resolution_url)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ky/bills.py", line 89, in scrape_bill_list
    yield from self.parse_bill(chamber, session, bill_id, link.attrib['href'])
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ky/bills.py", line 125, in parse_bill
    page = self.lxmlize(url)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/utils/lxmlize.py", line 24, in lxmlize
    response = self.get(url)
  File "/opt/**PGUSER**/venv-pupa/lib/python3.6/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/opt/**PGUSER**/venv-pupa/lib/python3.6/site-packages/scrapelib/__init__.py", line 292, in request
    raise HTTPError(resp)
scrapelib.HTTPError: 404 while retrieving https://apps.legislature.ky.gov/record/19RS/sr188.html
```

Visit http://bobsled.openstates.org for more info.



Discussion:
Working on this.

Resolving patch:
diff --git a/openstates/ky/bills.py b/openstates/ky/bills.py
--- a/openstates/ky/bills.py
+++ b/openstates/ky/bills.py
@@ -1,5 +1,6 @@
 import re
 import datetime
+import scrapelib
 from collections import defaultdict
 from pytz import timezone
 
@@ -122,7 +123,11 @@ def parse_bill_field(self, page, header):
         return page.xpath(xpath_expr)[0]
 
     def parse_bill(self, chamber, session, bill_id, url):
-        page = self.lxmlize(url)
+        try:
+            page = self.lxmlize(url)
+        except scrapelib.HTTPError as e:
+            self.logger.warning(e)
+            return
 
         last_action = self.parse_bill_field(
             page, 'Last Action').xpath('text()')[0]


Test patch: