Repository: openstates/openstates-scrapers

Issue:
PA failing since at least 2017-11-18
PA has been failing since 2017-11-18

Based on automated runs it appears that PA has not run successfully in 2 days (2017-11-18).


```
  23:09:19 INFO pupa: save person White, Martina A. as person_caf8a4fe-cce7-11e7-838a-0242ac11000a.json
23:09:19 INFO pupa: save membership caf8a4fe-cce7-11e7-838a-0242ac11000a membership in ~{"classification": "lower"} as membership_caf8a86e-cce7-11e7-838a-0242ac11000a.json
23:09:19 INFO pupa: save membership caf8a4fe-cce7-11e7-838a-0242ac11000a membership in ~{"classification": "party", "name": "Republican"} as membership_caf8aa9e-cce7-11e7-838a-0242ac11000a.json
23:09:19 INFO scrapelib: GET - http://www.legis.state.pa.us/cfdocs/legis/home/member_information/House_bio.cfm?id=264
23:09:21 INFO pupa: save person Youngblood, Rosita C. as person_cc2b3936-cce7-11e7-838a-0242ac11000a.json
23:09:21 INFO pupa: save membership cc2b3936-cce7-11e7-838a-0242ac11000a membership in ~{"classification": "lower"} as membership_cc2b3cc4-cce7-11e7-838a-0242ac11000a.json
23:09:21 INFO pupa: save membership cc2b3936-cce7-11e7-838a-0242ac11000a membership in ~{"classification": "party", "name": "Democratic"} as membership_cc2b3f12-cce7-11e7-838a-0242ac11000a.json
23:09:21 INFO scrapelib: GET - http://www.legis.state.pa.us/cfdocs/legis/home/member_information/House_bio.cfm?id=1711
23:09:23 INFO pupa: save person Zimmerman, David H. as person_cd5c2f0e-cce7-11e7-838a-0242ac11000a.json
23:09:23 INFO pupa: save membership cd5c2f0e-cce7-11e7-838a-0242ac11000a membership in ~{"classification": "lower"} as membership_cd5c3274-cce7-11e7-838a-0242ac11000a.json
23:09:23 INFO pupa: save membership cd5c2f0e-cce7-11e7-838a-0242ac11000a membership in ~{"classification": "party", "name": "Republican"} as membership_cd5c3512-cce7-11e7-838a-0242ac11000a.json
23:09:23 INFO pupa: no session specified, using 2017-2018
23:09:23 INFO scrapelib: GET - http://www.legis.state.pa.us/cfdocs/legis/bi/BillIndx.cfm?sYear=2017&sIndex=0&bod=S
23:09:24 INFO scrapelib: GET - http://www.legis.state.pa.us/cfdocs/billinfo/billinfo.cfm?syear=2017&sind=0&body=S&type=B&BN=1
23:09:25 INFO scrapelib: GET - http://www.legis.state.pa.us/cfdocs/billinfo/bill_history.cfm?syear=2017&sind=0&body=S&type=B&BN=1
23:09:27 INFO scrapelib: GET - http://www.legis.state.pa.us/cfdocs/billinfo/bill_votes.cfm?syear=2017&sind=0&body=S&type=B&bn=1
23:09:29 INFO scrapelib: GET - http://www.legis.state.pa.us/CFDOCS/Legis/RC/Public/rc_view_byBill.cfm?sess_yr=2017&sess_ind=0&rc_body=H&bill_body=S&bill_type=B&bill_nbr=1
23:09:31 INFO scrapelib: GET - http://www.legis.state.pa.us/CFDOCS/Legis/RC/Public/rc_view_byBill.cfm?sess_yr=2017&sess_ind=0&rc_body=H&bill_body=S&bill_type=B&bill_nbr=1
23:09:33 INFO scrapelib: GET - http://www.legis.state.pa.us/CFDOCS/Legis/RC/Public/rc_view_action2.cfm?sess_yr=2017&sess_ind=0&rc_body=H&rc_nbr=453
no pupa_settings on path, using defaults
pa (scrape, import)
  people: {}
  bills: {}
  committees: {}
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 67, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
Traceback (most recent call last):
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 111, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/openstates/openstates/openstates/pa/bills.py", line 29, in scrape
    yield from self.scrape_session(chamber, session)
  File "/opt/openstates/openstates/openstates/pa/bills.py", line 39, in scrape_session
    yield from self.parse_bill(chamber, session, special, link)
  File "/opt/openstates/openstates/openstates/pa/bills.py", line 77, in parse_bill
    utils.vote_url(chamber, session, special, type_abbr, bill_num),
  File "/opt/openstates/openstates/openstates/pa/bills.py", line 196, in parse_votes
    yield from self.parse_chamber_votes(bill, url)
  File "/opt/openstates/openstates/openstates/pa/bills.py", line 213, in parse_chamber_votes
    yield self.parse_roll_call(bill, link, chamber, date)
  File "/opt/openstates/openstates/openstates/pa/bills.py", line 235, in parse_roll_call
    yeas = int(page.xpath("//div[text() = 'YEAS']")[0].getnext().text)
IndexError: list index out of range
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/pa/bills.py b/openstates/pa/bills.py
--- a/openstates/pa/bills.py
+++ b/openstates/pa/bills.py
@@ -232,10 +232,24 @@ def parse_roll_call(self, bill, link, chamber, date):
             type = 'other'
             motion = link.text_content()
 
-        yeas = int(page.xpath("//div/span[text() = 'YEAS']/..")[0].getnext().text)
-        nays = int(page.xpath("//div/span[text() = 'NAYS']/..")[0].getnext().text)
-        lve = int(page.xpath("//div/span[text() = 'E']/..")[0].getnext().text)
-        nv = int(page.xpath("//div/span[text() = 'N/V']/..")[0].getnext().text)
+        # Looks like for "YEAS" and "NAYS" counts, PA has multiple HTML
+        # formats: one where the "YEAS" text node is nested within a span
+        # element, and another where the text node is a direct child of the div
+        # element
+        yeas_elements = page.xpath("//div/span[text() = 'YEAS']/..")
+        if len(yeas_elements) == 0:
+            yeas_elements = page.xpath("//div[text()[normalize-space() = 'YEAS']]")
+        yeas = int(yeas_elements[0].getnext().text)
+
+        nays_elements = page.xpath("//div/span[text() = 'NAYS']/..")
+        if len(nays_elements) == 0:
+            nays_elements = page.xpath("//div[text()[normalize-space() = 'NAYS']]")
+        nays = int(nays_elements[0].getnext().text)
+
+        # "LVE" and "N/V" have been moved up as direct children of the div
+        # element
+        lve = int(page.xpath("//div[text()[normalize-space() = 'LVE']]")[0].getnext().text)
+        nv = int(page.xpath("//div[text()[normalize-space() = 'N/V']]")[0].getnext().text)
         other = lve + nv
 
         vote = VoteEvent(


Test patch: