Repository: openstates/openstates-scrapers

Issue:
SD failing since at least 2018-07-25
SD has been failing since 2018-07-25

Based on automated runs it appears that SD has not run successfully in 2 days (2018-07-25).


```
  02:01:45 INFO pupa: save post 9 as post_c1c5c0f0-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 10 as post_c1c5c244-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 11 as post_c1c5c38e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 12 as post_c1c5c4d8-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 13 as post_c1c5c622-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 14 as post_c1c5c76c-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 15 as post_c1c5c8ac-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 16 as post_c1c5c9ec-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 17 as post_c1c5cb36-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 18 as post_c1c5cc76-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 19 as post_c1c5cdc0-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 20 as post_c1c5cf00-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 21 as post_c1c5d04a-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 22 as post_c1c5d18a-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 23 as post_c1c5d2d4-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 24 as post_c1c5d41e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 25 as post_c1c5d55e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 27 as post_c1c5d69e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 29 as post_c1c5d7e8-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 30 as post_c1c5d9f0-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 31 as post_c1c5dbb2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 32 as post_c1c5dcf2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 33 as post_c1c5de5a-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 34 as post_c1c5dfc2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 35 as post_c1c5e13e-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 26A as post_c1c5e292-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 26B as post_c1c5e3d2-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 28A as post_c1c5e508-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO pupa: save post 28B as post_c1c5e648-90a1-11e8-8ead-0244d764d5a8.json
02:01:45 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx
02:01:45 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/default.aspx
02:01:46 INFO scrapelib: GET - http://www.sdlegislature.gov/Legislators/default.aspx
loaded Open States pupa settings...
sd (scrape, import)
  people: {}
  bills: {}
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 120, in do_scrape
    raise ScrapeError('no objects returned from {} scrape'.format(self.__class__.__name__))
pupa.exceptions.ScrapeError: no objects returned from SDLegislatorScraper scrape
```

Visit http://bobsled.openstates.org for more info.



Discussion:
working on this.

Resolving patch:
diff --git a/openstates/sd/people.py b/openstates/sd/people.py
--- a/openstates/sd/people.py
+++ b/openstates/sd/people.py
@@ -7,17 +7,24 @@
 
 class SDLegislatorScraper(Scraper):
 
-    def scrape(self, chambers=None):
+    def scrape(self, session=None, chambers=None):
         self._committees = {}
 
+        if not session:
+            session = self.latest_session()
+            self.info('no session specified, using %s', session)
+
         # emails are on the contact page, fetch once and the
         # legislator scrapers can find their emails there
         contact_page_url = \
-            'http://www.sdlegislature.gov/Legislators/ContactLegislator.aspx'
+            'https://sdlegislature.gov/Legislators/ContactLegislator.aspx?Session={}'.format(
+                session)
         contact_page = self.get(contact_page_url).text
         contact_page = lxml.html.fromstring(contact_page)
 
-        url = 'http://www.sdlegislature.gov/Legislators/default.aspx'
+        # https://sdlegislature.gov/Legislators/default.aspx?Session=2018
+        url = 'https://sdlegislature.gov/Legislators/default.aspx?Session={}'.format(
+            session)
         if chambers is None:
             chambers = ['upper', 'lower']
         for chamber in chambers:
@@ -71,7 +78,8 @@ def scrape_legislator(self, name, chamber, url, contact_page):
                             )
         legislator.extras['occupation'] = occupation
         if office_phone.strip() != "":
-            legislator.add_contact_detail(type='voice', value=office_phone, note='Capitol Office')
+            legislator.add_contact_detail(
+                type='voice', value=office_phone, note='Capitol Office')
 
         # SD removed email from the detail pages but it's still in the
         # contact page, shared for all congress people
@@ -85,7 +93,8 @@ def scrape_legislator(self, name, chamber, url, contact_page):
             # look for the adjacent email mailto link
             profile_link = profile_link[0]
             profile_block = profile_link.getparent().getparent().getparent()
-            email_link = profile_block.xpath('./span/span/a[@class="mail-break"]')
+            email_link = profile_block.xpath(
+                './span/span/a[@class="mail-break"]')
             if email_link:
                 email = email_link[0].text
                 email = email.lstrip()
@@ -95,10 +104,10 @@ def scrape_legislator(self, name, chamber, url, contact_page):
                                                   value=email,
                                                   note='Capitol Office')
         home_address = [
-                x.strip() for x in
-                page.xpath('//td/span[contains(@id, "HomeAddress")]/text()')
-                if x.strip()
-                ]
+            x.strip() for x in
+            page.xpath('//td/span[contains(@id, "HomeAddress")]/text()')
+            if x.strip()
+        ]
         if home_address:
             home_address = "\n".join(home_address)
             home_phone = page.xpath(
@@ -114,7 +123,8 @@ def scrape_legislator(self, name, chamber, url, contact_page):
         legislator.add_source(url)
         legislator.add_link(url)
 
-        committees = page.xpath('//div[@id="divCommittees"]/span/section/table/tbody/tr/td/a')
+        committees = page.xpath(
+            '//div[@id="divCommittees"]/span/section/table/tbody/tr/td/a')
         for committee in committees:
             self.scrape_committee(legislator, url, committee, chamber)
         yield legislator
@@ -124,7 +134,8 @@ def scrape_committee(self, leg, url, element, chamber):
         if comm.startswith('Joint '):
             chamber = 'legislature'
 
-        role = element.xpath('../following-sibling::td')[0].text_content().lower().strip()
+        role = element.xpath(
+            '../following-sibling::td')[0].text_content().lower().strip()
 
         org = self.get_organization(comm, chamber)
         org.add_source(url)


Test patch: