Repository: openstates/openstates-scrapers

Issue:
IA failing since at least 2019-02-07
IA has been failing since 2019-02-07

Based on automated runs it appears that IA has not run successfully in 7 days (2019-02-07).


```
  06:19:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Bergan"}
06:19:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Mohr"}
06:19:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "James"}
06:19:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Nielsen"}
06:19:41 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Worthan"}
06:19:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Best"}
06:19:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Paustian"}
06:19:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Sieck"}
06:19:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Bloomingdale"}
06:19:42 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Thorup"}
06:19:43 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Sunde"}
06:19:44 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Deyoe"}
06:19:45 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Hall"}
06:19:45 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Judge"}
06:19:45 ERROR pupa: cannot resolve pseudo id to Person: ~{"name": "Matson"}
loaded Open States pupa settings...
ia (scrape, import)
  bills: {}
  votes: {}
On the question "Shall amendment H-1001 be adopted?" (H.F. 306)
On the question "Shall the bill pass?" (H.F. 306)
On the question "Shall the bill pass?" (H.F. 307)
On the question "Shall amendment H-1001 be adopted?" (H.F. 306)
On the question "Shall the bill pass?" (H.F. 306)
On the question "Shall the bill pass?" (H.F. 307)
import jurisdictions...
import bills...
import vote events...
Traceback (most recent call last):
  File "/opt/**PGUSER**/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 278, in handle
    return self.do_handle(args, other, juris)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 329, in do_handle
    report['import'] = self.do_import(juris, args)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 225, in do_import
    report.update(vote_event_importer.import_directory(datadir))
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/importers/base.py", line 196, in import_directory
    return self.import_data(json_stream())
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/importers/base.py", line 233, in import_data
    obj_id, what = self.import_item(data)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/importers/base.py", line 254, in import_item
    data = self.prepare_for_db(data)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/importers/vote_events.py", line 77, in prepare_for_db
    data['bill_id'] = self.bill_importer.resolve_json_id(bill)
  File "/opt/**PGUSER**/venv-pupa/src/pupa/pupa/importers/base.py", line 171, in resolve_json_id
    raise UnresolvedIdError(errmsg)
pupa.exceptions.UnresolvedIdError: cannot resolve pseudo id to Bill: ~{"from_organization__classification": "lower", "identifier": "HF 306", "legislative_session__identifier": "2019-2020"}
```

Visit http://bobsled.openstates.org for more info.



Discussion:
I took a look at it.  There have been a succession of problems.  The latest one just started in last night's run:
```
Max retries exceeded with url: /legislation/billTracking/billHistory?billName=HSB%20153&ga=88 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Try again',))
```
I wasn't able to reproduce it, and hope it is a transient state-side server issue.

The retry problem of yesterday went away, and we are left with the same problem seen in the original post.  From my run, it appears that several bills are not scraped, each console-logging a message like:

```WARNING pupa: URL: https://www.legis.iowa.gov/legislation/BillBook?ga=88&ba=SF 139 gives us an *EMPTY* bill. Aborting. ```

Failure to find these bills during the subsequent vote import causes the job abort.
Thanks @estaub i'll take a look at this today and see if either those bills shouldn't be skipped, or the votes should.
@showerst The title scrape is failing, in bills.py: 

```
        title = page.xpath('string(//div[@id="content"]/div[@class='
                           '"divideVert"]/div/div[4]/div[2])').strip()
```
@showerst FWIW, I'm stumped as to how that scrape EVER works.  For example, when I inspect bill https://www.legis.iowa.gov/legislation/BillBook?ga=88&ba=SF%20138 , from which the title
> A bill for an act relating to incentives for whole grade sharing and school district reorganization or dissolution. (Formerly SSB 1032.)

 is successfully scraped, I find no `div` with a `divideVert` class under the `content` `div`.
@estaub haha this took me a second too -- We're actually scraping a different page first, it's just not getting output into the scraper list. Look at line 67 of bills.py

It's not getting output to console because the original author did it with requests.get instead of self.get, probably so they could deal w/ those 500 errors cleanly.

Looks like the title on https://www.legis.iowa.gov/legislation/billTracking/billHistory?billName=SF%20139&ga=88 is in a different spot in the page structure than usual, so we're incorrectly skipping it. I'll see if i can whip up a fix. 
@showerst A small sample suggests that the structure changes when the bill passes.

The list at https://www.legis.iowa.gov/legislation/findLegislation/passedChamber?ga=88 has at least 3 bills I've seen blow up in bobsled.

Resolving patch:
diff --git a/openstates/ia/bills.py b/openstates/ia/bills.py
--- a/openstates/ia/bills.py
+++ b/openstates/ia/bills.py
@@ -81,8 +81,13 @@ def scrape_bill(self, chamber, session, session_id, bill_id, url):
                            '"divideVert"]/div/div[4]/div[2])').strip()
 
         if title == '':
-            self.warning("URL: %s gives us an *EMPTY* bill. Aborting." % url)
-            return
+            # Sometimes the title is moved, see
+            # https://www.legis.iowa.gov/legislation/billTracking/billHistory?billName=SF%20139&ga=88
+            title = page.xpath('string(//div[@id="content"]/div[@class='
+                               '"divideVert"]/div[4]/div[2])').strip()
+            if title == '':
+                self.warning("URL: %s gives us an *EMPTY* bill. Aborting." % url)
+                return
 
         if title.lower().startswith("in"):
             title = page.xpath("string(//table[2]/tr[3])").strip()


Test patch: