Repository: openstates/openstates-scrapers

Issue:
ND: Bill numbers are being added instead of vote names
The [ND vote scraper](http://github.com/openstates/openstates/blob/63ede9983ff6b8b26aeb2c29ad715f017c940746/openstates/nd/votes.py#L219) is saving bill numbers instead of the names of voters:

Some examples:
SB 2062, SB 2063, SB 2064, SB 2065, SB 2066, SB 2067

This is causing the vote to come back with a null leg_id, since the name cannot be matched back to a known legislator.



Discussion:
This is because the scraper assumes that the resulting verdict will all fit into one line, for example, "Engrossed SB 2201 passed".  If we look at[ the daily report](http://www.legis.nd.gov/assembly/65-2017/journals/sr-dailyjnl-17.pdf#page=3) for the day there was a large batch of bills voted on all at once, and thus this summary spilled over into multiple lines.  Not sure what the best fix for this is, but will try to get something working in the next couple days.
@mileswwatkins @jamesturk

The particular vote took place because the executive branch introduced 24 appropriations bills into the house and senate.  Given the legislative’s preference for their own appropriations bills they were batch rejected the executive ones using a consent calendar.

The person I talked to on the ND legislators council said he didn't recall this ever having been done in the past, certainly not in the past couple of years.

I think we have a couple options:
- Ignore the problem, and have the vote continue to only be identified with [SB 2085](https://openstates.org/nd/votes/NDV00007827/)
- Treat it as is 24 separate votes, each on one bill.
- Implement a many to one relationship with votes to bills.
This is tough!

**Ignore the problem**: Well, we know about it, so let's do something about it. We can fall back to this in _absolute worst_ case.
**Implement many to one**: We're too close to leaving Billy to do an implementation change like this on that end. @jamesturk, would OCD allow a 1:m between `Bill` and `VoteEvent`?
**Treat it as separate votes**: Might be the easiest option here, but I'm not sure how implementation would go, regarding assigning the vote to Billy `Vote` and `Bill` objects. Might have to fudge the "ID" of the vote?
Right now it is 1:1 in OCD.  My vote is for treating it as separate votes if this is indeed a weird exceptional case.  I think you'd just have to create 1 VoteEvent object per Bill, since bill is used as a differentiator of VoteEvent objects right now.
Alright, @NixGD, thanks for your gumshoe dedication. Can you modify the text parser to produce a discrete vote for each of the bills? Thanks!
Sounds like a plan!  This week has been a bit busy but I'll certainly fix it in a day or two.  Thanks for all your help!

Resolving patch:
diff --git a/openstates/nd/votes.py b/openstates/nd/votes.py
--- a/openstates/nd/votes.py
+++ b/openstates/nd/votes.py
@@ -38,6 +38,7 @@ def scrape(self, chamber, session):
             cur_vote = None
             in_vote = False
             cur_motion = ""
+            bills = []
 
             # Determine which URLs the information was pulled from
             pdf_url = pdf.attrib['href']
@@ -106,6 +107,7 @@ def scrape(self, chamber, session):
                         cur_vote = None
                         results = {}
                         cur_motion = ""
+                        bills = []
 
                     # If votes are being processed, record the voting members
                     elif ":" in line:
@@ -115,6 +117,14 @@ def scrape(self, chamber, session):
 
                         name_may_be_continued = False if line.endswith(";") \
                                 else True
+
+                    # Extracts bill numbers in the closing text
+                    # used for when the closing text is multiple lines.
+                    elif cur_vote is not None and\
+                            re.findall(r"(?i)(H|S|J)(C?)(B|R|M) (\d+)", line) and \
+                            not any(x in line.lower() for x in
+                            ['passed', 'adopted', 'sustained', 'prevailed', 'lost', 'failed']):
+                        bills.extend(re.findall(r"(?i)(H|S|J)(C?)(B|R|M) (\d+)", line))
                     
                     elif cur_vote is not None and \
                             not any(x in line.lower() for x in
@@ -139,7 +149,7 @@ def scrape(self, chamber, session):
 
                         # Identify what is being voted on
                         # Throw a warning if impropper informaiton found
-                        bills = re.findall(r"(?i)(H|S|J)(C?)(B|R|M) (\d+)", line)
+                        bills.extend(re.findall(r"(?i)(H|S|J)(C?)(B|R|M) (\d+)", line))
 
                         if bills == [] or cur_motion.strip() == "":
                             results = {}
@@ -151,8 +161,6 @@ def scrape(self, chamber, session):
                                     )
                             continue
 
-                        cur_bill_id = "%s%s%s %s" % (bills[-1])
-
                         # If votes are found in the motion name, throw an error
                         if "YEAS:" in cur_motion or "NAYS:" in cur_motion:
                             raise AssertionError(
@@ -184,61 +192,69 @@ def scrape(self, chamber, session):
                             "J": "joint"
                         }
 
-                        # Identify the source chamber for the bill
-                        try:
-                            bc = chambers[cur_bill_id[0]]
-                        except KeyError:
-                            bc = 'other'
-
-                        # Determine whether or not the vote passed
-                        if "over the governor's veto" in cur_motion.lower():
-                            VETO_SUPERMAJORITY = 2 / 3
-                            passed = (yes / (yes + no) > VETO_SUPERMAJORITY)
-                        else:
-                            passed = (yes > no)
-
-                        # Create a Vote object based on the scraped information
-                        vote = Vote(chamber,
-                                    cur_date,
-                                    cur_motion,
-                                    passed,
-                                    yes,
-                                    no,
-                                    other,
-                                    session=session,
-                                    bill_id=cur_bill_id,
-                                    bill_chamber=bc)
-
-                        vote.add_source(pdf_url)
-                        vote.add_source(url)
-
-                        # For each category of voting members,
-                        # add the individuals to the Vote object
-                        for key in res:
-                            obj = getattr(vote, key)
-                            for person in res[key]:
-                                obj(person)
-
-                        # Check the vote counts in the motion text against
-                        # the parsed results
-                        for category_name in keys.keys():
-                            # Need to search for the singular, not plural, in the text
-                            # so it can find, for example,  " 1 NAY "
-                            vote_re = r"(\d+)\s{}".format(category_name[:-1])
-                            motion_count = int(re.findall(vote_re, cur_motion)[0])
-                            vote_count = vote[keys[category_name] + "_count"]
-
-                            if motion_count != vote_count:
-                                self.warning(
-                                        "Motion text vote counts ({}) ".format(motion_count) +
-                                        "differed from roll call counts ({}) ".format(vote_count) +
-                                        "for {0} on {1}".format(category_name, cur_bill_id)
-                                        )
-                                vote[keys[category_name] + "_count"] = motion_count
-
-                        self.save_vote(vote)
+                        # Almost all of the time, a vote only applies to one bill and this loop will only be run once.
+                        # Some exceptions exist.
+
+                        for bill in bills:
+
+                            cur_bill_id = "%s%s%s %s" % bill
+
+                            # Identify the source chamber for the bill
+                            try:
+                                bc = chambers[cur_bill_id[0]]
+                            except KeyError:
+                                bc = 'other'
+
+                            # Determine whether or not the vote passed
+                            if "over the governor's veto" in cur_motion.lower():
+                                VETO_SUPERMAJORITY = 2 / 3
+                                passed = (yes / (yes + no) > VETO_SUPERMAJORITY)
+                            else:
+                                passed = (yes > no)
+
+                            # Create a Vote object based on the scraped information
+                            vote = Vote(chamber,
+                                        cur_date,
+                                        cur_motion,
+                                        passed,
+                                        yes,
+                                        no,
+                                        other,
+                                        session=session,
+                                        bill_id=cur_bill_id,
+                                        bill_chamber=bc)
+
+                            vote.add_source(pdf_url)
+                            vote.add_source(url)
+
+                            # For each category of voting members,
+                            # add the individuals to the Vote object
+                            for key in res:
+                                obj = getattr(vote, key)
+                                for person in res[key]:
+                                    obj(person)
+
+                            # Check the vote counts in the motion text against
+                            # the parsed results
+                            for category_name in keys.keys():
+                                # Need to search for the singular, not plural, in the text
+                                # so it can find, for example,  " 1 NAY "
+                                vote_re = r"(\d+)\s{}".format(category_name[:-1])
+                                motion_count = int(re.findall(vote_re, cur_motion)[0])
+                                vote_count = vote[keys[category_name] + "_count"]
+
+                                if motion_count != vote_count:
+                                    self.warning(
+                                            "Motion text vote counts ({}) ".format(motion_count) +
+                                            "differed from roll call counts ({}) ".format(vote_count) +
+                                            "for {0} on {1}".format(category_name, cur_bill_id)
+                                            )
+                                    vote[keys[category_name] + "_count"] = motion_count
+
+                            self.save_vote(vote)
 
                         # With the vote successfully processed,
                         # wipe its data and continue to the next one
                         results = {}
                         cur_motion = ""
+                        bills = []


Test patch: