Repository: openstates/openstates-scrapers

Issue:
DE failing since at least 2018-01-05
DE has been failing since 2018-01-05

Based on automated runs it appears that DE has not run successfully in 2 days (2018-01-05).


```
  23:12:44 INFO pupa: save bill HB 236 in 149 as bill_3883f630-f2a0-11e7-9ee7-0242ac110005.json
23:12:44 INFO scrapelib: GET - https://legis.delaware.gov/BillDetail?LegislationId=25982
23:12:45 INFO scrapelib: POST - https://legis.delaware.gov/json/BillDetail/GetRecentReportsByLegislationId
23:12:46 INFO scrapelib: POST - https://legis.delaware.gov/json/BillDetail/GetVotingReportsByLegislationId
23:12:47 INFO pupa: save bill HB 235 in 149 as bill_3a4e47e0-f2a0-11e7-9ee7-0242ac110005.json
23:12:47 INFO scrapelib: GET - https://legis.delaware.gov/BillDetail?LegislationId=25978
23:12:48 INFO scrapelib: POST - https://legis.delaware.gov/json/BillDetail/GetRecentReportsByLegislationId
23:12:49 INFO scrapelib: POST - https://legis.delaware.gov/json/BillDetail/GetVotingReportsByLegislationId
23:12:50 INFO pupa: save bill HB 234 in 149 as bill_3c187bd6-f2a0-11e7-9ee7-0242ac110005.json
23:12:50 INFO scrapelib: GET - https://legis.delaware.gov/BillDetail?LegislationId=25958
23:12:51 INFO scrapelib: POST - https://legis.delaware.gov/json/BillDetail/GetRecentReportsByLegislationId
23:12:52 INFO scrapelib: POST - https://legis.delaware.gov/json/BillDetail/GetVotingReportsByLegislationId
23:12:53 INFO pupa: save bill HB 232 in 149 as bill_3de299d8-f2a0-11e7-9ee7-0242ac110005.json
23:12:53 INFO scrapelib: GET - https://legis.delaware.gov/BillDetail?LegislationId=25947
no pupa_settings on path, using defaults
de (scrape, import)
  people: {}
  bills: {}
  committees: {}
SS 1 for SB 80
SS 1 for SB 76
SS 1 for SB 23
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
Traceback (most recent call last):
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 111, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/openstates/openstates/openstates/de/bills.py", line 32, in scrape
    yield from self.scrape_chamber(chamber, session)
  File "/opt/openstates/openstates/openstates/de/bills.py", line 39, in scrape_chamber
    yield from self.filter_substitutes(self.scrape_bill(row, chamber, session))
  File "/opt/openstates/openstates/openstates/de/bills.py", line 61, in filter_substitutes
    for bill in bills:
  File "/opt/openstates/openstates/openstates/de/bills.py", line 152, in scrape_bill
    self.scrape_fiscal_note(bill, fiscal)
  File "/opt/openstates/openstates/openstates/de/bills.py", line 188, in scrape_fiscal_note
    bill.add_document_link('Fiscal Note', fn_url, media_type='text/html')
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/bill.py", line 106, in add_document_link
    on_duplicate=on_duplicate)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 299, in _add_associated_link
    raise ScrapeValueError("Duplicate entry in '%s' - URL: '%s'" % (collection, url))
pupa.exceptions.ScrapeValueError: Duplicate entry in 'documents' - URL: 'https://legis.delaware.gov/json/BillDetail/'
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/de/bills.py b/openstates/de/bills.py
--- a/openstates/de/bills.py
+++ b/openstates/de/bills.py
@@ -177,15 +177,8 @@ def scrape_legislators(self, session):
             self.warning("Error returning legislator list!")
 
     def scrape_fiscal_note(self, bill, link):
-        # https://legis.delaware.gov/json/BillDetail/GetHtmlDocument?fileAttachmentId=48095
-        # The DE site for some reason POSTS to an endpoint for fiscal notes
-        # then pops the JSON response into a new window.
-        # But the attachment endpoint for bill versions works fine.
-        attachment_id = link.replace('GetFiscalNoteHtmlDocument(event, ', '').replace(')', '')
-        fn_url = 'https://legis.delaware.gov/json/BillDetail/'
-        'GetHtmlDocument?fileAttachmentId={}'.format(attachment_id)
-        # on_duplicate='error'
-        bill.add_document_link('Fiscal Note', fn_url, media_type='text/html')
+        media_type = self.mime_from_link(link)
+        bill.add_document_link('Fiscal Note', link, media_type=media_type)
 
     def scrape_votes(self, bill, legislation_id, session):
         votes_url = 'https://legis.delaware.gov/json/BillDetail/GetVotingReportsByLegislationId'


Test patch: