Repository: openstates/openstates-scrapers

Issue:
ME failing since at least 2018-04-04
ME has been failing since 2018-04-04

Based on automated runs it appears that ME has not run successfully in 2 days (2018-04-04).


```
  02:50:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "GUERIN"}
02:50:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "GRIGNON"}
02:50:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "SEAVEY"}
02:50:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "HILL"}
02:50:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "McLEAN"}
02:50:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "SHEATS"}
02:50:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "DUCHESNE"}
02:50:43 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "HIGGINS"}
02:50:44 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "BICKFORD"}
02:50:45 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "LOCKMAN"}
02:50:46 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "PERKINS"}
02:50:47 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "WHITE"}
02:50:48 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "DANA"}
02:50:48 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "BEAR"}
02:50:48 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "BRADSTREET"}
02:50:48 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "HAGGAN"}
02:50:48 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "REED"}
02:50:48 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "STROM"}
02:50:57 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Mason"}
loaded Open States pupa settings...
me (scrape, import)
  bills: {}
  people: {}
  committees: {}
import jurisdictions...
import organizations...
import people...
import posts...
import memberships...
import bills...
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 307, in do_handle
    report['import'] = self.do_import(juris, args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 213, in do_import
    report.update(bill_importer.import_directory(datadir))
  File "/opt/openstates/venv-pupa/src/pupa/pupa/importers/base.py", line 196, in import_directory
    return self.import_data(json_stream())
  File "/opt/openstates/venv-pupa/src/pupa/pupa/importers/base.py", line 233, in import_data
    obj_id, what = self.import_item(data)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/importers/base.py", line 272, in import_item
    raise DuplicateItemError(data, obj, related.get('sources', []))
pupa.exceptions.DuplicateItemError: attempt to import data that would conflict with data already in the import: {'classification': ['bill'], 'identifier': 'LD 1616', 'extras': {'ld_number': 'LD 1616'}, 'subject': ['STATUTES', 'ERRORS AND INCONSISTENCIES', '128TH 1ST REGULAR'], 'from_organization_id': 'ocd-organization/a26314f0-dc6f-4cfc-99ca-dee4bafc9d4c', 'legislative_session_id': UUID('3fdcf437-7a5a-4abe-820d-52578b42a7ef'), 'title': 'An Act To Correct Errors And Inconsistencies In The Laws Of Maine'} (already imported as LD 1616 in Maine 128th Legislature (2017-2018))
obj1 sources: ['http://legislature.maine.gov/LawMakerWeb/subjects.asp?ID=280068396', 'http://legislature.maine.gov/LawMakerWeb/dockets.asp?ID=280068396', 'http://legislature.maine.gov/LawMakerWeb/sponsors.asp?ID=280068396', 'http://legislature.maine.gov/LawMakerWeb/summary.asp?ID=280068396']
obj2 sources: ['http://legislature.maine.gov/LawMakerWeb/summary.asp?ID=280065483', 'http://legislature.maine.gov/LawMakerWeb/sponsors.asp?ID=280065483', 'http://legislature.maine.gov/LawMakerWeb/dockets.asp?ID=280065483', 'http://legislature.maine.gov/LawMakerWeb/subjects.asp?ID=280065483']
```

Visit http://bobsled.openstates.org for more info.



Discussion:
Looking at it...
Yeah.  So the bill has two semi-duplicate pages, one good:
    http://legislature.maine.gov/LawMakerWeb/summary.asp?ID=280065483
and one less good:
    http://legislature.maine.gov/LawMakerWeb/summary.asp?ID=280068396

The only difference between them, at least in the JSON we scrape, is that the good one has all the actions, the bad one has none.

I don't feel comfortable with it, but I'm planning to just put in an URL filter to ignore the bad one.  I think it's probably valid to have a bill with no actions.

Resolving patch:
diff --git a/openstates/me/bills.py b/openstates/me/bills.py
--- a/openstates/me/bills.py
+++ b/openstates/me/bills.py
@@ -72,6 +72,8 @@ def _recursively_process_bills(
         if bills:
             for bill in bills:
                 bill_id_slug = bill.xpath('./@href')[0]
+                if bill_id_slug == 'summary.asp?ID=280068396':
+                    continue
                 bill_url = 'http://legislature.maine.gov/LawMakerWeb/{}'.format(bill_id_slug)
                 bill_id = bill.text[:2] + " " + bill.text[2:]
 


Test patch: