Repository: openstates/openstates-scrapers

Issue:
KS failing since at least 2018-03-23
KS has been failing since 2018-03-23

Based on automated runs it appears that KS has not run successfully in 2 days (2018-03-23).


```
  01:42:28 INFO pupa: save membership 856dd784-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "party", "name": "Republican"} as membership_856dde78-2f2e-11e8-827d-0a58a9feac2a.json
01:42:28 INFO scrapelib: GET - http://www.kslegislature.org/li/api/v11/rev-1/members/rep_trimboli_frank_1
01:42:44 INFO scrapelib: GET - http://www.kslegislature.org/li/b2017_18/members/rep_trimboli_frank_1/
01:42:44 INFO pupa: save person Frank Trimboli as person_8eab65dc-2f2e-11e8-827d-0a58a9feac2a.json
01:42:44 INFO pupa: save membership 8eab65dc-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "lower"} as membership_8eab6a28-2f2e-11e8-827d-0a58a9feac2a.json
01:42:44 INFO pupa: save membership 8eab65dc-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "party", "name": "Republican"} as membership_8eab6ca8-2f2e-11e8-827d-0a58a9feac2a.json
01:42:44 INFO scrapelib: GET - http://www.kslegislature.org/li/api/v11/rev-1/members/rep_curtis_pam_1
01:42:59 INFO scrapelib: GET - http://www.kslegislature.org/li/b2017_18/members/rep_curtis_pam_1/
01:42:59 INFO pupa: save person Pam Curtis as person_97e07746-2f2e-11e8-827d-0a58a9feac2a.json
01:42:59 INFO pupa: save membership 97e07746-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "lower"} as membership_97e07b74-2f2e-11e8-827d-0a58a9feac2a.json
01:42:59 INFO pupa: save membership 97e07746-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "party", "name": "Democratic"} as membership_97e07dea-2f2e-11e8-827d-0a58a9feac2a.json
01:42:59 INFO scrapelib: GET - http://www.kslegislature.org/li/api/v11/rev-1/members/rep_eplee_john_1
01:43:05 INFO scrapelib: GET - http://www.kslegislature.org/li/b2017_18/members/rep_eplee_john_1/
01:43:09 INFO pupa: save person John Eplee as person_9dd7e7d8-2f2e-11e8-827d-0a58a9feac2a.json
01:43:09 INFO pupa: save membership 9dd7e7d8-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "lower"} as membership_9dd7ec10-2f2e-11e8-827d-0a58a9feac2a.json
01:43:09 INFO pupa: save membership 9dd7e7d8-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "party", "name": "Republican"} as membership_9dd7ee86-2f2e-11e8-827d-0a58a9feac2a.json
01:43:09 INFO scrapelib: GET - http://www.kslegislature.org/li/api/v11/rev-1/members/rep_bergquist_emil_1
01:43:15 INFO scrapelib: GET - http://www.kslegislature.org/li/b2017_18/members/rep_bergquist_emil_1/
01:43:19 INFO pupa: save person Emil Bergquist as person_a3cf23b8-2f2e-11e8-827d-0a58a9feac2a.json
01:43:19 INFO pupa: save membership a3cf23b8-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "lower"} as membership_a3cf27fa-2f2e-11e8-827d-0a58a9feac2a.json
01:43:19 INFO pupa: save membership a3cf23b8-2f2e-11e8-827d-0a58a9feac2a membership in ~{"classification": "party", "name": "Republican"} as membership_a3cf2a7a-2f2e-11e8-827d-0a58a9feac2a.json
01:43:19 INFO pupa: no session specified, using 2017-2018
01:43:19 INFO scrapelib: GET - http://www.kslegislature.org/li/api/v11/rev-1/bill_status/
loaded Open States pupa settings...
ks (scrape, import)
  committees: {}
  people: {}
  bills: {}
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 111, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/openstates/openstates/openstates/ks/bills.py", line 25, in scrape
    yield from self.scrape_chamber(chamber, session)
  File "/opt/openstates/openstates/openstates/ks/bills.py", line 31, in scrape_chamber
    bill_request = self.get(ksapi.url + 'bill_status/').text
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/scrapelib/__init__.py", line 292, in request
    raise HTTPError(resp)
scrapelib.HTTPError: 503 while retrieving http://www.kslegislature.org/li/api/v11/rev-1/bill_status/
```

Visit http://bobsled.openstates.org for more info.



Discussion:
Looking into it.
Its giving 503(temporary overload or scheduled maintenance) status code, on browser too, I am not sure whether to wait since its been 4 days.
http://www.kslegislature.org/li/api/v11/rev-1/bill_status/ is accessible now, 

but scrape of http://www.kslegislature.org/li/b2017_18/measures/vote_view/je_20180329192334_686100/ is failing with a new problem, scraping date of vote.  

Most likely, the page has been reformatted.


I'm working this one; I'll either PR or post a giving-up post-mortem here when done.

Resolving patch:
diff --git a/openstates/ks/bills.py b/openstates/ks/bills.py
--- a/openstates/ks/bills.py
+++ b/openstates/ks/bills.py
@@ -186,12 +186,14 @@ def scrape_html(self, bill, session):
     def parse_vote(self, bill, link):
         member_doc = lxml.html.fromstring(self.get(link).text)
         motion = member_doc.xpath("//div[@id='main_content']/h4/text()")
-        opinions = member_doc.xpath("//div[@id='main_content']/h3/text()")
+        chamber_date_line = ''.join(member_doc.xpath("//div[@id='main_content']/h3[1]//text()"))
+        chamber_date_line_words = chamber_date_line.split()
+        vote_chamber = chamber_date_line_words[0]
+        vote_date = datetime.datetime.strptime(chamber_date_line_words[-1], '%m/%d/%Y')
+        vote_status = " ".join(chamber_date_line_words[2:-2])
+
+        opinions = member_doc.xpath("//div[@id='main_content']/h3[position() > 1]/text()")
         if len(opinions) > 0:
-            temp = opinions[0].split()
-            vote_chamber = temp[0]
-            vote_date = datetime.datetime.strptime(temp[-1], '%m/%d/%Y')
-            vote_status = " ".join(temp[2:-2])
             vote_status = vote_status if vote_status.strip() else motion[0]
             vote_chamber = 'upper' if vote_chamber == 'Senate' else 'lower'
 


Test patch: