Repository: openstates/openstates-scrapers

Issue:
MN failing since at least 2018-01-11
MN has been failing since 2018-01-11

Based on automated runs it appears that MN has not run successfully in 2 days (2018-01-11).


```
  01:04:04 INFO pupa: save membership c66b3da8-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Democratic-Farmer-Labor"} as membership_c66b42f8-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save person David J. Osmek as person_c66b97f8-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66b97f8-f766-11e7-8096-0242ac110003 membership in ~{"classification": "upper"} as membership_c66b9b0e-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66b97f8-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Republican"} as membership_c66b9d34-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save person Sandra L. Pappas as person_c66bfc3e-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66bfc3e-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Democratic-Farmer-Labor"} as membership_c66c01a2-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66bfc3e-f766-11e7-8096-0242ac110003 membership in ~{"classification": "upper"} as membership_c66bff72-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save person Eric R. Pratt as person_c66c54ea-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66c54ea-f766-11e7-8096-0242ac110003 membership in ~{"classification": "upper"} as membership_c66c580a-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66c54ea-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Republican"} as membership_c66c5a3a-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66cae4a-f766-11e7-8096-0242ac110003 membership in ~{"classification": "upper"} as membership_c66cb16a-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save person Jerry Relph as person_c66cae4a-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66cae4a-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Republican"} as membership_c66cb39a-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save person Ann H. Rest as person_c66d0728-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66d0728-f766-11e7-8096-0242ac110003 membership in ~{"classification": "upper"} as membership_c66d0a48-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66d0728-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Democratic-Farmer-Labor"} as membership_c66d0c6e-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save person Julie A. Rosen as person_c66d666e-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66d666e-f766-11e7-8096-0242ac110003 membership in ~{"classification": "upper"} as membership_c66d69a2-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66d666e-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Republican"} as membership_c66d6bd2-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save person Carrie Ruud as person_c66dc118-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66dc118-f766-11e7-8096-0242ac110003 membership in ~{"classification": "upper"} as membership_c66dc438-f766-11e7-8096-0242ac110003.json
01:04:04 INFO pupa: save membership c66dc118-f766-11e7-8096-0242ac110003 membership in ~{"classification": "party", "name": "Republican"} as membership_c66dc690-f766-11e7-8096-0242ac110003.json
  people: {}
  committees: {}
no pupa_settings on path, using defaults
  vote_events: {}
  bills: {}
mn (scrape, import)
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/spatula/base.py", line 11, in scrape_page_items
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 111, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/openstates/openstates/openstates/mn/people.py", line 143, in scrape
    yield from self.scrape_page_items(SenList)
KeyError: 'image'
    yield from page_type(self, url=url, **kwargs).handle_page()
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/spatula/base.py", line 121, in handle_page
    processed = self.handle_list_item(item)
  File "/opt/openstates/openstates/openstates/mn/people.py", line 57, in handle_list_item
    image=self.extra_info[name]['image'])
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/mn/people.py b/openstates/mn/people.py
--- a/openstates/mn/people.py
+++ b/openstates/mn/people.py
@@ -16,6 +16,7 @@
 class SenList(CSV):
     url = 'http://www.senate.mn/members/member_list_ascii.php?ls='
     _html_url = 'http://www.senate.mn/members/index.php'
+    danSchoenSeen = False
 
     def __init__(self, scraper, url=None, *, obj=None, **kwargs):
         super().__init__(scraper, url=url, obj=obj, **kwargs)
@@ -50,6 +51,10 @@ def _scrape_extra_info(self):
     def handle_list_item(self, row):
         if not row['First Name']:
             return
+        # This Dan Schoen hack very probably can be removed after April 2018.
+        if row['First Name'] == 'Dan' and row['Last Name'] == 'Schoen':
+            self.danSchoenSeen = True
+            return
         name = '{} {}'.format(row['First Name'], row['Last Name'])
         party = PARTIES[row['Party']]
         leg = Person(name=name, district=row['District'].lstrip('0'),
@@ -94,6 +99,12 @@ def handle_list_item(self, row):
 
         return leg
 
+    def handle_page(self):
+        yield super(SenList, self).handle_page()
+        if not self.danSchoenSeen:
+            raise AssertionError('Dan Schoen not seen, remove his kludge')
+            # https://github.com/openstates/openstates/pull/2088
+
 
 class RepList(Page):
     url = 'http://www.house.leg.state.mn.us/members/hmem.asp'


Test patch: