Repository: openstates/openstates-scrapers

Issue:
MA failing since at least 2019-06-09
MA has been failing since 2019-06-09

Based on automated runs it appears that MA has not run successfully in 2 days (2019-06-09).


```
  07:43:08 INFO pupa: save bill H351 in 191st as bill_63afe646-8b53-11e9-b118-023bcd60a39c.json
07:43:08 INFO scrapelib: GET - https://malegislature.gov/Bills/191/H3670
07:43:10 INFO pupa: GET (with `requests`) - https://malegislature.gov/Bills/191/H3670
07:43:11 INFO pupa: save bill H3670 in 191st as bill_6523d334-8b53-11e9-b118-023bcd60a39c.json
07:43:11 INFO scrapelib: GET - https://malegislature.gov/Bills/191/H2796
07:43:13 INFO pupa: GET (with `requests`) - https://malegislature.gov/Bills/191/H2796
07:43:13 INFO pupa: save bill H2796 in 191st as bill_66c1fdb0-8b53-11e9-b118-023bcd60a39c.json
07:43:13 INFO scrapelib: GET - https://malegislature.gov/Bills/191/H2670
07:43:16 INFO pupa: GET (with `requests`) - https://malegislature.gov/Bills/191/H2670
07:43:16 INFO pupa: save bill H2670 in 191st as bill_686ae9c4-8b53-11e9-b118-023bcd60a39c.json
07:43:16 INFO scrapelib: GET - https://malegislature.gov/Bills/191/H3811
07:43:18 INFO pupa: GET (with `requests`) - https://malegislature.gov/Bills/191/H3811
07:43:19 INFO pupa: save bill H3811 in 191st as bill_69d83a6e-8b53-11e9-b118-023bcd60a39c.json
07:43:19 INFO scrapelib: GET - https://malegislature.gov/Bills/191/H1209
07:43:21 INFO pupa: GET (with `requests`) - https://malegislature.gov/Bills/191/H1209
07:43:21 INFO pupa: save bill H1209 in 191st as bill_6b90b354-8b53-11e9-b118-023bcd60a39c.json
07:43:21 INFO scrapelib: GET - https://malegislature.gov/Bills/191/H86
07:43:24 INFO pupa: GET (with `requests`) - https://malegislature.gov/Bills/191/H86
loaded Open States pupa settings...
ma (scrape, import)
  bills: {}
Traceback (most recent call last):
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ma/bills.py", line 279, in scrape_action_page
    y, n = re.search(r'(\d+) yeas .*? (\d+) nays', action_name.lower()).groups()
AttributeError: 'NoneType' object has no attribute 'groups'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/**PGUSER**/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 278, in handle
    return self.do_handle(args, other, juris)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 327, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 175, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/scrape/base.py", line 112, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ma/bills.py", line 61, in scrape
    yield from self.scrape_chamber('lower', session)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ma/bills.py", line 80, in scrape_chamber
    yield from self.scrape_bill(session, bill, chamber)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ma/bills.py", line 182, in scrape_bill
    self.scrape_actions(bill, bill_url, session)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ma/bills.py", line 212, in scrape_actions
    self.scrape_action_page(bill, page)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/ma/bills.py", line 283, in scrape_action_page
    y = int(re.search(r"yeas\s+(\d+)", action_name.lower()).group(1))
AttributeError: 'NoneType' object has no attribute 'group'
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/ma/bills.py b/openstates/ma/bills.py
--- a/openstates/ma/bills.py
+++ b/openstates/ma/bills.py
@@ -275,32 +275,37 @@ def scrape_action_page(self, bill, page):
                 actor = "upper"
                 # placeholder
                 vote_action = action_name.split(' -')[0]
-                try:
-                    y, n = re.search(r'(\d+) yeas .*? (\d+) nays', action_name.lower()).groups()
-                    y = int(y)
-                    n = int(n)
-                except AttributeError:
-                    y = int(re.search(r"yeas\s+(\d+)", action_name.lower()).group(1))
-                    n = int(re.search(r"nays\s+(\d+)", action_name.lower()).group(1))
-
-                # TODO: other count isn't included, set later
-                cached_vote = VoteEvent(
-                    chamber=actor,
-                    start_date=action_date,
-                    motion_text=vote_action,
-                    result='pass' if y > n else 'fail',
-                    classification='passage',
-                    bill=bill,
-                )
-                cached_vote.set_count('yes', y)
-                cached_vote.set_count('no', n)
-
-                rollcall_pdf = 'http://malegislature.gov' + row.xpath('string(td[3]/a/@href)')
-                self.scrape_senate_vote(cached_vote, rollcall_pdf)
-                cached_vote.add_source(rollcall_pdf)
-                cached_vote.pupa_id = rollcall_pdf
-                # XXX: also disabled, see above note
-                # yield cached_vote
+                # 2019 H86 Breaks our regex,
+                # Ordered to a third reading --
+                # see Senate   Roll Call #25 and House Roll Call 56
+                if 'yeas' in action_name and 'nays' in action_name:
+                    try:
+                        y, n = re.search(r'(\d+) yeas .*? (\d+) nays',
+                                         action_name.lower()).groups()
+                        y = int(y)
+                        n = int(n)
+                    except AttributeError:
+                        y = int(re.search(r"yeas\s+(\d+)", action_name.lower()).group(1))
+                        n = int(re.search(r"nays\s+(\d+)", action_name.lower()).group(1))
+
+                    # TODO: other count isn't included, set later
+                    cached_vote = VoteEvent(
+                        chamber=actor,
+                        start_date=action_date,
+                        motion_text=vote_action,
+                        result='pass' if y > n else 'fail',
+                        classification='passage',
+                        bill=bill,
+                    )
+                    cached_vote.set_count('yes', y)
+                    cached_vote.set_count('no', n)
+
+                    rollcall_pdf = 'http://malegislature.gov' + row.xpath('string(td[3]/a/@href)')
+                    self.scrape_senate_vote(cached_vote, rollcall_pdf)
+                    cached_vote.add_source(rollcall_pdf)
+                    cached_vote.pupa_id = rollcall_pdf
+                    # XXX: also disabled, see above note
+                    # yield cached_vote
 
             attrs = self.categorizer.categorize(action_name)
             action = bill.add_action(


Test patch: