Repository: openstates/openstates-scrapers

Issue:
ID: scrape fails on substitute representative
Once #2053 is merged, the people scraper is blowing up reading https://legislature.idaho.gov/house/membership/ .  
Search down to "Scanlin", who you'll find is a "substitute" representative.  There's a parsing problem because the scraper expects the name to be bold and the party to not be; in this case "Substitute" is bold only.

For a description of what a "Substitute" is, see http://www.spokesman.com/blogs/boise/2014/mar/16/idahos-legislative-substitute-law-unique/ 

Just fixing this would leave two legislators in that district, where, as in Highlander, "there can be only one".  Representative Kloc, who Scanlin is substituting for, is directly above him.

Assuming that his roll call votes, committee memberships, etc. are registered to the substitute, I believe a fully correct fix for this would involve:
- lookahead for a "Substitute", skipping the current legislator if the next legislator is a substitute.
- rewriting the line parser to skip "Substitute" and just use the text (not bold format) for name/party parsing.



Discussion:
we never came up w/ a great canonical way to handle substitutes

it maybe feels like in this case we should allow two reps to hold that seat for the time being, thoughts @mileswwatkins ? 
Hrrrrrrmmmmmmm.

My instinct would be to ignore the substitutes, as if they didn't exist. On a practical side, as the article points out, sometimes people sub in for periods as short as 3 days. I don't think that Open States should change its representation of who's sitting in a district for such a short period. On a conceptual side, "who's my legislator" should point to the _elected_ official in my mind, not the person who's substituting in for them.

Thoughts, @jamesturk and @estaub?
yeah that makes sense, it's frustrating though as it'll almost always lead
to unresolved votes.

but yeah, let's ignore subs

On Jan 12, 2018 16:49, "Miles Watkins" <notifications@github.com> wrote:

Hrrrrrrmmmmmmm.

My instinct would be to ignore the substitutes, as if they didn't exist. On
a practical side, as the article points out, sometimes people sub in for
periods as short as 3 days. I don't think that Open States should change
its representation of who's sitting in a district for such a short period.
On a conceptual side, "who's my legislator" should point to the *elected*
official in my mind, not the person who's substituting in for them.

Thoughts, @jamesturk <https://github.com/jamesturk> and @estaub
<https://github.com/estaub>?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
<https://github.com/openstates/openstates/issues/2082#issuecomment-357363835>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAfYg9VrlG1XERfp_-e8_a11HPsasRWks5tJ9NXgaJpZM4Rckns>
.

I'm good.  
In the best of all worlds, it _might_ be good to map the substitute's votes/memberships onto the elected leg, as if they were there all the time.

Resolving patch:
diff --git a/openstates/id/people.py b/openstates/id/people.py
--- a/openstates/id/people.py
+++ b/openstates/id/people.py
@@ -78,7 +78,8 @@ def scrape_chamber(self, chamber):
             img_url = row.xpath('.//img/@src')[0]
 
             inner = row.xpath('.//div[@class="vc-column-innner-wrapper"]')[1]
-            if 'Resigned' in inner.text_content():
+            inner_text = inner.text_content()
+            if 'Resigned' in inner_text or 'Substitute' in inner_text:
                 continue
 
             name = inner.xpath('p/strong')[0].text.replace(u'\xa0', ' ').strip()


Test patch: