Repository: openstates/openstates-scrapers

Issue:
OH failing since at least 2018-04-18
OH has been failing since 2018-04-18

Based on automated runs it appears that OH has not run successfully in 2 days (2018-04-18).


```
  02:46:44 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb602/disapprove
02:46:45 INFO pupa: save bill HB 602 in 132 as bill_cb814bda-43a5-11e8-a12a-020dd716e63e.json
02:46:45 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/
02:46:46 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/actions
02:46:47 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/votes
02:46:48 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/cmtevotes
02:46:49 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/veto
02:46:50 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/disapprove
02:46:51 INFO pupa: save bill HB 603 in 132 as bill_cf14d0f0-43a5-11e8-a12a-020dd716e63e.json
02:46:51 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/
02:46:52 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/actions
02:46:53 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/votes
02:46:54 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/cmtevotes
02:46:55 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/veto
02:46:56 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/disapprove
02:46:57 INFO pupa: save bill HB 604 in 132 as bill_d2b4f550-43a5-11e8-a12a-020dd716e63e.json
02:46:57 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/
02:46:58 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/actions
02:46:59 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/votes
02:47:00 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/cmtevotes
02:47:01 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/veto
02:47:02 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/disapprove
02:47:03 INFO pupa: save bill HB 605 in 132 as bill_d6465452-43a5-11e8-a12a-020dd716e63e.json
02:47:03 INFO scrapelib: GET - http://www.ohiosenate.gov/members/senate-directory
loaded Open States pupa settings...
oh (scrape, import)
  bills: {}
  people: {}
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 111, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/openstates/openstates/openstates/oh/people.py", line 48, in scrape
    yield from self.scrape_page('upper', CHAMBER_URLS['upper'])
  File "/opt/openstates/openstates/openstates/oh/people.py", line 131, in scrape_page
    page = self.get(url).text
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/scrapelib/__init__.py", line 292, in request
    raise HTTPError(resp)
scrapelib.HTTPError: 404 while retrieving http://www.ohiosenate.gov/members/senate-directory
```

Visit http://bobsled.openstates.org for more info.



Discussion:
Looks like a site redesign; there could be a lot more problems.  The member directory is now at http://www.ohiosenate.gov/senators ; I haven't tried to figure out how it compares.
Yes, its a redesign. For senators at least. @estaub if you are not working on this one, I will take it up.

@In-vincible - Take it away!

Resolving patch:
diff --git a/openstates/oh/people.py b/openstates/oh/people.py
--- a/openstates/oh/people.py
+++ b/openstates/oh/people.py
@@ -32,7 +32,7 @@
 }
 
 CHAMBER_URLS = {
-    'upper': "http://www.ohiosenate.gov/members/senate-directory",
+    'upper': "http://www.ohiosenate.gov/senators",
     'lower':  "http://www.ohiohouse.gov/members/member-directory"
 }
 
@@ -45,8 +45,8 @@ def scrape(self, chamber=None):
         if chamber:
             yield from self.scrape_page(chamber, CHAMBER_URLS[chamber])
         else:
-            yield from self.scrape_page('upper', CHAMBER_URLS['upper'])
-            yield from self.scrape_page('lower', CHAMBER_URLS['lower'])
+            yield from self.scrape_senator_page('upper', CHAMBER_URLS['upper'])
+            yield from self.scrape_member_page('lower', CHAMBER_URLS['lower'])
         yield from self.committees.values()
 
     def fetch_committee_positions(self, a):
@@ -127,7 +127,7 @@ def scrape_homepage(self, leg, chamber, homepage):
             org.add_source(homepage)
             leg.add_membership(org)
 
-    def scrape_page(self, chamber, url):
+    def scrape_member_page(self, chamber, url):
         page = self.get(url).text
         page = lxml.html.fromstring(page)
         page.make_links_absolute(url)
@@ -183,3 +183,44 @@ def scrape_page(self, chamber, url):
             leg.add_source(url)
             leg.add_link(homepage)
             yield leg
+
+    def scrape_senator_page(self, chamber, url):
+        page = self.get(url).text
+        page = lxml.html.fromstring(page)
+        page.make_links_absolute(url)
+
+        for legislator in page.xpath("//div[contains(concat(' ', normalize-space(@class), ' '), "
+                                     "' portraitContainer ')]"):
+            img = legislator.xpath(".//div[@class='profileThumbnailBoundingBox']/@style")[0]
+            img = img[img.find('(')+1:img.find(')')]
+            full_name = legislator.xpath(".//div[@class='profileName']/a/text()")[0]
+            homepage_url = legislator.xpath(".//a[@class='profileImageLink']")[0].attrib['href']
+            district = legislator.xpath(".//div[@class='profileDistrict']"
+                                        "/a/text()")[0].split("#")[1]
+
+            if "Vacant" in full_name:
+                continue
+
+            homepage = self.get(homepage_url).text
+            page = lxml.html.fromstring(homepage)
+            phone = page.xpath("//div[@class='phone']/span/text()")[0]
+
+            address_lines = page.xpath("//div[@class='address']/span/text()")
+            address = "\n".join(address_lines)
+
+            email = (
+                'rep{0:0{width}}@ohiohouse.gov'
+                if chamber == 'lower' else
+                'sd{0:0{width}}@ohiosenate.gov'
+            ).format(int(district), width=2)
+
+            leg = Person(name=full_name, district=district,
+                         primary_org=chamber, image=img)
+
+            leg.add_contact_detail(type='address', value=address, note='Capitol Office')
+            leg.add_contact_detail(type='voice', value=phone, note='Capitol Office')
+            leg.add_contact_detail(type='email', value=email, note='Capitol Office')
+
+            leg.add_source(url)
+            leg.add_link(homepage_url)
+            yield leg


Test patch: