Repository: openstates/openstates-scrapers

Issue:
NY: No 2017 bills contain any Assembly votes
There are separate issues (#1384 , #1831) about missing roll calls.  This is about there being no 2017 Assembly votes at all in any bills.  I presume the other issues will remain after this is fixed, hence the new issue.


Discussion:
Example:
https://openstates.org/ny/bills/2017-2018/A40001/#votes

Source vote:
http://nyassembly.gov/leg/?default_fld=&leg_video=&bn=A40001&term=&Summary=Y&Floor%26nbspVotes=Y
to clarify, how is this distinct from the two tickets you cited? 
The other issues pre-existed, and are about individual legislator roll call yea/nay's being missing.  This issue is about there being no Assembly votes at all under any bills.  

I'm guessing that these have different causes, since this one started happening later.
AFAICT, there is no code for scraping Assembly votes.  The Billy code was not migrated to Pupa.  @jmcarp, do you recall whether there are issues with the NY source data that make Assembly vote scraping impossible or prohibitively difficult?  See [this commit](https://github.com/openstates/openstates/commit/c29e88790b3f462c854539de9bc7144928ed2517#diff-c0b833d744907b858a610c3983ecd2c8), file `models.py`,  to see the stripped code.  

finally got a chance to look at this, NY Assembly bill scraping is definitely broken, we need to remove the whole ny/models.py thing that isn't working and add a new scraper
Can I work on this issue?
@jmcarp Why did you remove the `_build_*` methods from `models.py` in [this commit](https://github.com/openstates/openstates/commit/c29e88790b3f462c854539de9bc7144928ed2517#diff-c0b833d744907b858a610c3983ecd2c8L58)?

Resolving patch:
diff --git a/openstates/ny/bills.py b/openstates/ny/bills.py
--- a/openstates/ny/bills.py
+++ b/openstates/ny/bills.py
@@ -273,7 +273,7 @@ def _scrape_bill(self, session, bill_data):
                 yield self._parse_senate_votes(vote_data, bill, api_url)
         elif bill_chamber == 'lower':
             assembly = AssemblyBillPage(self, session, bill, details)
-            assembly.build()
+            yield from assembly.build()
 
         # A little strange the way it works out, but the Assembly
         # provides the HTML version documents and the Senate provides
diff --git a/openstates/ny/models.py b/openstates/ny/models.py
--- a/openstates/ny/models.py
+++ b/openstates/ny/models.py
@@ -2,8 +2,11 @@
 import inspect
 
 from openstates.utils import LXMLMixin
+from pupa.scrape import VoteEvent
 
 from .utils import Urls
+import datetime
+from pytz import timezone
 
 
 class AssemblyBillPage(LXMLMixin):
@@ -54,4 +57,71 @@ def build(self):
         for name, member in inspect.getmembers(self):
             if inspect.ismethod(member):
                 if name.startswith('_build_'):
-                    member()
+                    yield member()
+
+    def _build_lower_votes(self):
+        url = self.shared_url + '&Floor%26nbspVotes=Y'
+        self.urls.add(votes=url)
+        self.bill.add_source(url)
+        doc = self.urls.votes.doc
+        if doc is None:
+            return
+
+        # Grab bill information.
+        try:
+            pre = doc.xpath('//pre')[0].text_content().strip()
+
+            no_votes = 'There are no votes for this bill in this legislative '
+
+            if pre == no_votes:
+                raise ValueError('No votes for this bill.')
+        # Skip bill if votes can't be found.
+        except (IndexError, ValueError) as e:
+            return
+
+        for table in doc.xpath('//table'):
+
+            date = table.xpath('caption/span[contains(., "DATE:")]')
+            date = next(date[0].itersiblings()).text
+            date = datetime.datetime.strptime(date, '%m/%d/%Y')
+            date = date.replace(tzinfo=timezone('UTC'))
+
+            votes = table.xpath('caption/span/span')[0].text.split(':')[1].split('/')
+            yes_count, no_count = map(int, votes)
+            passed = yes_count > no_count
+            vote = VoteEvent(
+                chamber='lower',
+                start_date=date,
+                motion_text='Floor Vote',
+                bill=self.bill,
+                result='pass' if passed else 'fail',
+                classification='passage'
+            )
+
+            vote.set_count('yes', yes_count)
+            vote.set_count('no', no_count)
+            absent_count = 0
+            excused_count = 0
+            tds = table.xpath('tr/td/text()')
+            votes = [tds[i:i+2] for i in range(0, len(tds), 2)]
+
+            vote_dictionary = {
+                'Y': 'yes',
+                'NO': 'no',
+                'ER': 'excused',
+                'AB': 'absent',
+                'NV': 'not voting'
+            }
+
+            for vote_pair in votes:
+                name, vote_val = vote_pair
+                vote.vote(vote_dictionary[vote_val], name)
+                if vote_val == 'AB':
+                    absent_count += 1
+                elif vote_val == 'ER':
+                    excused_count += 1
+
+            vote.set_count('absent', absent_count)
+            vote.set_count('excused', excused_count)
+            vote.add_source(url)
+            yield vote
diff --git a/openstates/ny/utils.py b/openstates/ny/utils.py
--- a/openstates/ny/utils.py
+++ b/openstates/ny/utils.py
@@ -45,7 +45,7 @@ def __repr__(self):
     @CachedAttr
     def text(self):
         text = self.scraper.get(self.url).text
-        self.urls_object.validate(self.name, self.url, text)
+        # self.urls_object.validate(self.name, self.url, text)
         return text
 
     @CachedAttr


Test patch: