Repository: openstates/openstates-scrapers

Issue:
NH Bill text points to wrong document.
State: NH

Many or all bill texts for NH's 2017 session are wrong.  Use the openstates.org website to look at a few.
In case the problem isn't as obvious as I think, here's one:

NH's `HB 544` (OpenStates `NHB00006021`) is a bill titled: 

> relative to earned time credits for prisoners participating in rehabilitative or educational programming.

A single version is listed in the bill detail API response, with a URL of 
http://www.gencourt.state.nh.us/bill_status/billText.aspx?sy=2017&id=0761&txtFormat=html

The document at that URL is a fiscal note on an unrelated bill, `HB 407`.

I have not tried to determine if this affects earlier sessions.


Discussion:
The problem seems to be an off-by-one error in the `id` query parameter in the document URL.

In the case above, where the bill detail API returns
http://www.gencourt.state.nh.us/bill_status/billText.aspx?sy=2017&id=0761&txtFormat=html
it should actually be returning
http://www.gencourt.state.nh.us/bill_status/billText.aspx?sy=2017&id=760&txtFormat=html

where the only difference is the id being 761 and 760, respectively.

I suspect it's more complicated that an off-by-one error most of the time. 80% of the time the bill's NH internal ID (LSR) works in that slot, but then if there was another doc published  around the same time, sometimes the ID gets off.

I'm going to look at this tonight, but in case anyone beats me to it we just need to find the mapping between the LSR and the &id= to drop into billText.aspx. 

If someone can check the text files http://gencourt.state.nh.us/dynamicdatafiles/ for any row that maps either the bill number HB 544 or LSR 761 to "760" that should do it.
N.B. - The mapping is in http://gencourt.state.nh.us/dynamicdatafiles/LsrsOnly.txt?x=20170124121048 

I'll put in a pull request tonight for this.

Resolving patch:
diff --git a/openstates/nh/bills.py b/openstates/nh/bills.py
--- a/openstates/nh/bills.py
+++ b/openstates/nh/bills.py
@@ -59,6 +59,11 @@ def scrape(self, chamber, session):
         # bill basics
         self.bills = {}         # LSR->Bill
         self.bills_by_id = {}   # need a second table to attach votes
+        self.versions_by_lsr = {} # mapping of bill ID to lsr
+
+        # pre load the mapping table of LSR -> version id
+        self.scrape_version_ids()
+
         last_line = []
         for line in self.get('http://gencourt.state.nh.us/dynamicdatafiles/LSRs.txt').content.split("\n"):
             line = line.split('|')
@@ -90,7 +95,7 @@ def scrape(self, chamber, session):
                 elif expanded_bill_id.startswith('PET'):
                     bill_type = 'petition'
                 elif expanded_bill_id.startswith('AR') and bill_id.startswith('CACR'):
-                    bill_type = 'constitutional amendment'    
+                    bill_type = 'constitutional amendment'
                 else:
                     bill_type = bill_type_map[expanded_bill_id.split(' ')[0][1:]]
 
@@ -100,11 +105,16 @@ def scrape(self, chamber, session):
                 self.bills[lsr] = Bill(session, chamber, bill_id, title,
                                        type=bill_type)
 
-                # http://www.gencourt.state.nh.us/bill_status/billText.aspx?sy=2017&id=95&txtFormat=html                       
-                version_url = 'http://www.gencourt.state.nh.us/bill_status/billText.aspx?sy={}&id={}&txtFormat=html'.format(session, lsr)
+                # http://www.gencourt.state.nh.us/bill_status/billText.aspx?sy=2017&id=95&txtFormat=html
+                if lsr in self.versions_by_lsr:
+                    version_id = self.versions_by_lsr[lsr]
+                    version_url = 'http://www.gencourt.state.nh.us/bill_status/' \
+                                  'billText.aspx?sy={}&id={}&txtFormat=html' \
+                                  .format(session, version_id)
+
+                    self.bills[lsr].add_version('latest version', version_url,
+                                                mimetype='text/html', on_duplicate='use_new')
 
-                self.bills[lsr].add_version('latest version', version_url,
-                                            mimetype='text/html')
                 self.bills_by_id[bill_id] = self.bills[lsr]
 
         # load legislators
@@ -178,6 +188,21 @@ def add_source(self, bill, lsr, session):
         bill_url = 'http://www.gencourt.state.nh.us/bill_Status/bill_status.aspx?lsr={}&sy={}&sortoption=&txtsessionyear={}'.format(lsr, session, session)
         bill.add_source(bill_url)
 
+    def scrape_version_ids(self):
+        for line in self.get('http://gencourt.state.nh.us/dynamicdatafiles/LsrsOnly.txt').content.split("\n"):
+            if len(line) < 1:
+                continue
+            # a few blank/irregular lines, irritating
+            if '|' not in line:
+                continue
+
+            line = line.split('|')
+            file_id = line[2]
+            lsr = line[0].split('-')
+            lsr = lsr[1]
+            self.versions_by_lsr[lsr] = file_id
+
+
     def scrape_votes(self, session):
         votes = {}
         last_line = []


Test patch: