Repository: openstates/openstates-scrapers

Issue:
NY: No 2017 bills contain any Assembly votes
There are separate issues (#1384 , #1831) about missing roll calls.  This is about there being no 2017 Assembly votes at all in any bills.  I presume the other issues will remain after this is fixed, hence the new issue.


Discussion:
Example:
https://openstates.org/ny/bills/2017-2018/A40001/#votes

Source vote:
http://nyassembly.gov/leg/?default_fld=&leg_video=&bn=A40001&term=&Summary=Y&Floor%26nbspVotes=Y
to clarify, how is this distinct from the two tickets you cited? 
The other issues pre-existed, and are about individual legislator roll call yea/nay's being missing.  This issue is about there being no Assembly votes at all under any bills.  

I'm guessing that these have different causes, since this one started happening later.
AFAICT, there is no code for scraping Assembly votes.  The Billy code was not migrated to Pupa.  @jmcarp, do you recall whether there are issues with the NY source data that make Assembly vote scraping impossible or prohibitively difficult?  See [this commit](https://github.com/openstates/openstates/commit/c29e88790b3f462c854539de9bc7144928ed2517#diff-c0b833d744907b858a610c3983ecd2c8), file `models.py`,  to see the stripped code.  

finally got a chance to look at this, NY Assembly bill scraping is definitely broken, we need to remove the whole ny/models.py thing that isn't working and add a new scraper
Can I work on this issue?
@jmcarp Why did you remove the `_build_*` methods from `models.py` in [this commit](https://github.com/openstates/openstates/commit/c29e88790b3f462c854539de9bc7144928ed2517#diff-c0b833d744907b858a610c3983ecd2c8L58)?

Resolving patch:
diff --git a/openstates/ny/bills.py b/openstates/ny/bills.py
--- a/openstates/ny/bills.py
+++ b/openstates/ny/bills.py
@@ -274,6 +274,7 @@ def _scrape_bill(self, session, bill_data):
         elif bill_chamber == 'lower':
             assembly = AssemblyBillPage(self, session, bill, details)
             assembly.build()
+            bill = assembly.bill
 
         # A little strange the way it works out, but the Assembly
         # provides the HTML version documents and the Senate provides
diff --git a/openstates/ny/models.py b/openstates/ny/models.py
--- a/openstates/ny/models.py
+++ b/openstates/ny/models.py
@@ -1,8 +1,12 @@
+import collections
+import datetime
 import re
 import inspect
 
-from openstates.utils import LXMLMixin
+from itertools import islice
 
+from billy.scrape.votes import Vote
+from openstates.utils import LXMLMixin
 from .utils import Urls
 
 
@@ -34,20 +38,83 @@ def __init__(self, scraper, session, bill, details):
             'senate': senate_url})
 
     def _scrub_name(self, name):
-        junk = [
+        junk = (re.compile(rgx) for rgx in [
             r'^Rules\s+',
             '\(2nd Vice Chairperson\)',
             '\(MS\)',
             'Assemblyman',
             'Assemblywoman',
-            'Senator']
+            'Senator'])
         for rgx in junk:
-            name = re.sub(rgx, '', name, re.I)
+            name = rgx.sub('', name, re.I)
 
         # Collapse whitespace.
         name = re.sub('\s+', ' ', name)
         return name.strip('(), ')
 
+    def _build_sponsors_memo(self):
+        url = self.shared_url + '&Memo=Y'
+        self.bill.add_document('Sponsor\'s Memorandum', url)
+
+    def _build_lower_votes(self):
+        url = self.shared_url + '&Floor%26nbspVotes=Y'
+        self.urls.add(votes=url)
+        self.bill.add_source(url)
+        doc = self.urls.votes.doc
+        if doc is None:
+            return
+
+        # Grab bill information.
+        try:
+            pre = doc.xpath('//pre')[0].text_content().strip()
+
+            no_votes = 'There are no votes for this bill in this legislative '
+            'session.'
+
+            if pre == no_votes:
+                raise ValueError('No votes for this bill.')
+        # Skip bill if votes can't be found.
+        except (IndexError, ValueError) as e:
+            return
+
+        actual_vote = collections.defaultdict(list)
+        for table in doc.xpath('//table'):
+
+            date = table.xpath('caption/span[contains(., "DATE:")]')
+            date = next(date[0].itersiblings()).text
+            date = datetime.datetime.strptime(date, '%m/%d/%Y')
+
+            votes = table.xpath('caption/span/span')[0].text.split(':')[1].split('/')
+            yes_count, no_count = map(int, votes)
+
+            passed = yes_count > no_count
+            vote = Vote('lower', date, 'Floor Vote', passed, yes_count,
+                        no_count, other_count=0)
+
+            tds = table.xpath('tr/td/text()')
+            votes = iter(tds)
+            while True:
+                try:
+                    data = list(islice(votes, 2))
+                    name, vote_val = data
+                except (StopIteration, ValueError):
+                    # End of data. Stop.
+                    break
+                name = self._scrub_name(name)
+
+                if vote_val.strip() == 'Y':
+                    vote.yes(name)
+                elif vote_val.strip() in ('N', 'NO'):
+                    vote.no(name)
+                else:
+                    vote.other(name)
+                    actual_vote[vote_val].append(name)
+
+            # The page doesn't provide an other_count.
+            vote['other_count'] = len(vote['other_votes'])
+            vote['actual_vote'] = actual_vote
+            self.bill.add_vote(vote)
+
     def build(self):
         '''Run all the build_* functions.
         '''


Test patch: