Repository: openstates/openstates-scrapers

Issue:
IL failing since at least 2019-04-05
IL has been failing since 2019-04-05

Based on automated runs it appears that IL has not run successfully in 2 days (2019-04-05).


```
  08:47:52 INFO pupa: save bill HB2693 in 101st as bill_a9ae13c2-5848-11e9-b796-02e2b4b1c1e0.json
08:47:52 INFO scrapelib: GET - http://ilga.gov/legislation/votehistory.asp?DocNum=2693&DocTypeID=HB&LegID=118902&GAID=15&SessionID=108&GA=101&SpecSess=
08:47:53 INFO scrapelib: GET - http://ilga.gov/legislation/votehistory/101/house/committeevotes/10100HB2693_24328.pdf
08:47:54 INFO pupa: save vote_event 101st - 2019-03-20 - Do Pass as vote_event_ab749c44-5848-11e9-b796-02e2b4b1c1e0.json
08:47:54 INFO scrapelib: GET - http://ilga.gov/legislation/BillStatus.asp?DocNum=2694&GAID=15&DocTypeID=HB&LegId=118904&SessionID=108&GA=101
08:47:55 INFO scrapelib: GET - http://ilga.gov/legislation/fulltext.asp?DocName=&SessionId=108&GA=101&DocTypeId=HB&DocNum=2694&GAID=15&LegID=118904&SpecSess=&Session=
08:47:56 INFO pupa: save bill HB2694 in 101st as bill_ac0f96cc-5848-11e9-b796-02e2b4b1c1e0.json
08:47:56 INFO scrapelib: GET - http://ilga.gov/legislation/votehistory.asp?DocNum=2694&DocTypeID=HB&LegID=118904&GAID=15&SessionID=108&GA=101&SpecSess=
08:47:57 INFO scrapelib: GET - http://ilga.gov/legislation/BillStatus.asp?DocNum=2695&GAID=15&DocTypeID=HB&LegId=118905&SessionID=108&GA=101
08:47:58 INFO scrapelib: GET - http://ilga.gov/legislation/fulltext.asp?DocName=&SessionId=108&GA=101&DocTypeId=HB&DocNum=2695&GAID=15&LegID=118905&SpecSess=&Session=
08:47:59 INFO pupa: save bill HB2695 in 101st as bill_add8df90-5848-11e9-b796-02e2b4b1c1e0.json
08:47:59 INFO scrapelib: GET - http://ilga.gov/legislation/votehistory.asp?DocNum=2695&DocTypeID=HB&LegID=118905&GAID=15&SessionID=108&GA=101&SpecSess=
08:48:00 INFO scrapelib: GET - http://ilga.gov/legislation/BillStatus.asp?DocNum=2696&GAID=15&DocTypeID=HB&LegId=118906&SessionID=108&GA=101
08:48:01 INFO scrapelib: GET - http://ilga.gov/legislation/fulltext.asp?DocName=&SessionId=108&GA=101&DocTypeId=HB&DocNum=2696&GAID=15&LegID=118906&SpecSess=&Session=
08:48:02 INFO pupa: save bill HB2696 in 101st as bill_afa365ca-5848-11e9-b796-02e2b4b1c1e0.json
08:48:02 INFO scrapelib: GET - http://ilga.gov/legislation/votehistory.asp?DocNum=2696&DocTypeID=HB&LegID=118906&GAID=15&SessionID=108&GA=101&SpecSess=
08:48:03 INFO scrapelib: GET - http://ilga.gov/legislation/BillStatus.asp?DocNum=2697&GAID=15&DocTypeID=HB&LegId=118907&SessionID=108&GA=101
08:48:04 INFO scrapelib: GET - http://ilga.gov/legislation/fulltext.asp?DocName=&SessionId=108&GA=101&DocTypeId=HB&DocNum=2697&GAID=15&LegID=118907&SpecSess=&Session=
08:48:05 INFO pupa: save bill HB2697 in 101st as bill_b16da794-5848-11e9-b796-02e2b4b1c1e0.json
08:48:05 INFO scrapelib: GET - http://ilga.gov/legislation/votehistory.asp?DocNum=2697&DocTypeID=HB&LegID=118907&GAID=15&SessionID=108&GA=101&SpecSess=
08:48:06 INFO scrapelib: GET - http://ilga.gov/legislation/BillStatus.asp?DocNum=2698&GAID=15&DocTypeID=HB&LegId=118910&SessionID=108&GA=101
08:48:07 WARNING pupa: [HB2698] Couldn't find sponsor [lower,Robert Rita] to refine
08:48:07 INFO scrapelib: GET - http://ilga.gov/legislation/fulltext.asp?DocName=&SessionId=108&GA=101&DocTypeId=HB&DocNum=2698&GAID=15&LegID=118910&SpecSess=&Session=
loaded Open States pupa settings...
il (scrape, import)
  bills: {}
Traceback (most recent call last):
  File "/opt/**PGUSER**/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 278, in handle
    return self.do_handle(args, other, juris)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 327, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 175, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/scrape/base.py", line 112, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/il/bills.py", line 268, in scrape
    yield from self.scrape_bill(chamber, session_id, doc_type, bill_url)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/il/bills.py", line 364, in scrape_bill
    self.scrape_documents(bill, version_url)
  File "/opt/**PGUSER**/**PGUSER**/**PGUSER**/il/bills.py", line 394, in scrape_documents
    bill.add_version_link(name, url, media_type=mimetype)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/scrape/bill.py", line 112, in add_version_link
    on_duplicate=on_duplicate)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/scrape/base.py", line 303, in _add_associated_link
    raise ScrapeValueError("Duplicate entry in '%s' - URL: '%s'" % (collection, url))
pupa.exceptions.ScrapeValueError: Duplicate entry in 'versions' - URL: 'http://ilga.gov/legislation/101/HB/PDF/10100HB2698lv.pdf'
```

Visit http://bobsled.openstates.org for more info.



Discussion:
when a version link is causing an issue, the easiest fix is to add on_duplicate='ignore' to the add_version_link call, after verifying that this isn't an issue w/ the scraper
Looking more closely at [that bill](http://ilga.gov/legislation/billstatus.asp?DocNum=2698&GAID=15&GA=101&DocTypeID=HB&LegID=118910&SessionID=108), looks like the bill versions are:

### Introduced
* http://ilga.gov/legislation/fulltext.asp?DocName=10100HB2698&GA=101&SessionId=108&DocTypeId=HB&LegID=118910&DocNum=2698&GAID=15&Session=
* http://ilga.gov/legislation/101/HB/PDF/10100HB2698.pdf

### Engrossed
*  http://ilga.gov/legislation/fulltext.asp?DocName=10100HB2698eng&GA=101&SessionId=108&DocTypeId=HB&LegID=118910&DocNum=2698&GAID=15&Session=
http://ilga.gov/legislation/101/HB/PDF/10100HB2698eng.pdf

But Open States API returns:

```
{
  "data": {
    "bill": {
      "id": "ocd-bill/c27b5d57-e91e-4740-84c9-c9f126996238",
      "title": "$AUDITOR GEN",
      "versions": [
        {
          "note": "Introduced",
          "date": "",
          "links": [
            {
              "mediaType": "application/pdf",
              "url": "http://ilga.gov/legislation/101/HB/PDF/10100HB2698lv.pdf",
              "text": ""
            }
          ]
        }
      ]
    }
  }
}
```
So it seems like maybe there is some incorrect logic (or else a chronological change that caused bad data).

The 10100HB2698lv.pdf file *does* correspond to the PDF link on the page that first appears when clicking ["Full text"](http://ilga.gov/legislation/fulltext.asp?DocName=&SessionId=108&GA=101&DocTypeId=HB&DocNum=2698&GAID=15&LegID=118910&SpecSess=&Session=), but the text content of that file is identical to the engrossed file (10100HB2698eng.pdf). They are two distinct files, there is a *slight* visual difference.

At first glance it's not clear what the "lv" version is supposed to be. Will investigate more.

Resolving patch:
diff --git a/openstates/il/bills.py b/openstates/il/bills.py
--- a/openstates/il/bills.py
+++ b/openstates/il/bills.py
@@ -383,22 +383,32 @@ def scrape_documents(self, bill, version_url):
         for link in doc.xpath('//a[contains(@href, "fulltext")]'):
             name = link.text
             url = link.get('href')
-            url = '{}&print=true'.format(url)
-            mimetype = 'text/html'
-            if name in VERSION_TYPES:
-                if pdf_only:
-                    pdf_link = doc.xpath('//a[text()="PDF"]')[0]
-                    url = pdf_link.get('href')
-                    mimetype = 'application/pdf'
-
-                bill.add_version_link(name, url, media_type=mimetype)
-            elif 'Amendment' in name or name in FULLTEXT_DOCUMENT_TYPES:
-                bill.add_document_link(name, url)
-            elif 'Printer-Friendly' in name:
-                pass
-            else:
-                self.warning('unknown document type %s - adding as document' % name)
-                bill.add_document_link(name, url)
+
+            # Ignore the "Printer-friendly version" link
+            # That link is a "latest version" alias for an actual, distinct version
+            if 'print=true' not in url:
+                if name in VERSION_TYPES:
+                    if pdf_only:
+                        # eed to visit the version's page, and get PDF link from there
+                        # otherwise get a faulty "latest version"/"LV" alias/duplicate
+                        version_page_html = self.get(url).text
+                        version_page_doc = lxml.html.fromstring(version_page_html)
+                        version_page_doc.make_links_absolute(url)
+                        pdf_link = version_page_doc.xpath('//a[text()="PDF"]')[0]
+                        url = pdf_link.get('href')
+                        mimetype = 'application/pdf'
+                    else:
+                        url = '{}&print=true'.format(url)
+                        mimetype = 'text/html'
+
+                    bill.add_version_link(name, url, media_type=mimetype)
+                elif 'Amendment' in name or name in FULLTEXT_DOCUMENT_TYPES:
+                    bill.add_document_link(name, url)
+                elif 'Printer-Friendly' in name:
+                    pass
+                else:
+                    self.warning('unknown document type %s - adding as document' % name)
+                    bill.add_document_link(name, url)
 
     def scrape_votes(self, session, bill, votes_url, committee_actors):
         html = self.get(votes_url).text


Test patch: