Repository: openstates/openstates-scrapers

Issue:
DE failing since at least 2017-12-08
DE has been failing since 2017-12-08

Based on automated runs it appears that DE has not run successfully in 2 days (2017-12-08).


```
  23:16:51 WARNING pupa: skipping amendment HA 1 to HJR 5
23:16:51 WARNING pupa: skipping amendment HA 1 to HB 4
23:16:51 WARNING pupa: skipping amendment HA 2 to HCR 39
23:16:52 WARNING pupa: skipping amendment HA 5 to SS 1 for SB 5
23:16:52 WARNING pupa: skipping amendment HA 1 to HS 1 for HB 85
23:16:52 WARNING pupa: skipping amendment HA 2 to HS 1 for HCR 2
23:16:52 WARNING pupa: skipping amendment HA 1 to HS 1 for HCR 2
23:16:52 WARNING pupa: skipping amendment HA 1 to HS 1 for HB 173
23:16:52 WARNING pupa: skipping amendment HA 1 to HS 1 for HB 180
23:16:52 WARNING pupa: skipping amendment HA 2 to HS 1 for HB 180
23:16:52 WARNING pupa: skipping amendment HA 3 to HS 1 for HB 180
23:16:52 WARNING pupa: skipping amendment HA 1 to HA 2 to HS 1 for HB 180
23:16:52 WARNING pupa: skipping amendment HA 1 to HS 1 for HB 143
23:16:52 WARNING pupa: skipping amendment HA 1 to HS 1 for HB 22
23:16:52 WARNING pupa: skipping amendment HA 4 to SS 1 for SB 5
23:16:52 WARNING pupa: skipping amendment HA 1 to HCR 2
no pupa_settings on path, using defaults
de (scrape, import)
  people: {}
  bills: {}
  committees: {}
HS 1 for HB 36
SS 1 for SB 76
SS 1 for SB 23
HS 1 for HB 179
HS 1 for HB 42
HS 1 for HB 12
SS 1 for SB 80
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
TypeError: argument of type 'NoneType' is not iterable
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 111, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/openstates/openstates/openstates/de/bills.py", line 32, in scrape
    yield from self.scrape_chamber(chamber, session)
  File "/opt/openstates/openstates/openstates/de/bills.py", line 48, in scrape_chamber
    yield from self.filter_substitutes(self.scrape_bill(row, chamber, session))
  File "/opt/openstates/openstates/openstates/de/bills.py", line 61, in filter_substitutes
    for bill in bills:
  File "/opt/openstates/openstates/openstates/de/bills.py", line 78, in scrape_bill
    if 'Substituted' in row['StatusName']:
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 67, in main
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/de/bills.py b/openstates/de/bills.py
--- a/openstates/de/bills.py
+++ b/openstates/de/bills.py
@@ -75,7 +75,14 @@ def filter_substitutes(self, bills):
     def scrape_bill(self, row, chamber, session):
         bill_id = row['LegislationDisplayCode']
 
-        if 'Substituted' in row['StatusName']:
+        # hack for empty StatusName
+        statusless_bills = ['HA 2 to SS 1 for SB 5', 'HA 3 to SS 1 for SB 5']
+        is_force_substitute = bill_id in statusless_bills \
+            and row['StatusName'] is None
+
+        is_substituted = is_force_substitute or 'Substituted' in row['StatusName'] \
+
+        if is_substituted:
             # skip substituted bills, the replacement is picked up instead
             self.warning('skipping %s: %s', bill_id, row['StatusName'])
             return


Test patch: