internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: openstates/openstates-scrapers

Issue:
IN: leading 0 in url link to senate bill
In Indiana senate bills such as this one:
https://openstates.org/in/bills/2017/SB285/

the url provided to the actual bill contains a leading zero:
http://iga.in.gov/legislative/2017/bills/senate/0285

The bill id has a leading 0 but the working URL does not contain the leading zeros:
http://iga.in.gov/legislative/2017/bills/senate/285


Discussion:

Resolving patch:
diff --git a/openstates/in/bills.py b/openstates/in/bills.py
--- a/openstates/in/bills.py
+++ b/openstates/in/bills.py
@@ -24,7 +24,7 @@ class INBillScraper(BillScraper):
 
     def _get_bill_id_components(self, bill_id):
         bill_prefix = ''.join([c for c in bill_id if c.isalpha()])
-        bill_number = ''.join([c for c in bill_id if c.isdigit()])
+        bill_number = ''.join([c for c in bill_id if c.isdigit()]).lstrip('0')
 
         return (bill_prefix, bill_number)
 


Test patch:
Repository: openstates/openstates-scrapers

Issue:
NH: legislators scrape is failing
State: NH

Legislator scraper needs some attention


Discussion:

Resolving patch:
diff --git a/openstates/nh/legislators.py b/openstates/nh/legislators.py
--- a/openstates/nh/legislators.py
+++ b/openstates/nh/legislators.py
@@ -38,13 +38,13 @@ def _parse_legislator(self, row, chamber, term):
         # Capture legislator vitals.
         first_name = row['FirstName']
         middle_name = row['MiddleName']
-        last_name = row['lastname']
+        last_name = row['LastName']
         full_name = '{} {} {}'.format(first_name, middle_name, last_name)
         full_name = re.sub(r'[\s]{2,}', ' ', full_name)
 
-        district = '{} {}'.format(row['county'], int(row['District'])).strip()
-        party = self.party_map[row['party']]
-        email = row['EMailAddress1']
+        district = '{} {}'.format(row['County'], int(row['District'])).strip()
+        party = self.party_map[row['party'].upper()]
+        email = row['WorkEmail']
 
         legislator = Legislator(term, chamber, district, full_name,
                                 first_name=first_name, last_name=last_name,
@@ -52,8 +52,8 @@ def _parse_legislator(self, row, chamber, term):
                                 email=email)
 
         # Capture legislator office contact information.
-        district_address = '{}\n{}\n{}, {} {}'.format(row['street'],
-            row['address2'], row['city'], row['state'], row['zipcode']).strip()
+        district_address = '{}\n{}\n{}, {} {}'.format(row['Address'],
+            row['address2'], row['city'], row['State'], row['Zipcode']).strip()
 
         legislator.add_office('district', 'Home Address',
                               address=district_address)


Test patch:
Repository: openstates/openstates-scrapers

Issue:
WV: subcommittees treated as full committees
Hello,

So, with WV, the subcommittees (e.g., Finance subcommittees A, B, C) are not identified as a subcommittee.

I know WV is a [round C state](https://github.com/openstates/planning/issues/11), but I thought it was worth mentioning...if only for a "TODO later" note.

Best,
Alex


Discussion:

Resolving patch:
diff --git a/openstates/wv/committees.py b/openstates/wv/committees.py
--- a/openstates/wv/committees.py
+++ b/openstates/wv/committees.py
@@ -8,6 +8,14 @@
 class WVCommitteeScraper(CommitteeScraper):
     jurisdiction = "wv"
 
+    # Manually resolved links between subcommittees and parent committees.
+    subcommittee_parent_map = {
+        'Post Audits Subcommittee': 'Government and Finance',
+        'Parks, Recreation and Natural Resources Subcommittee': \
+            'Government and Finance',
+        'Tax Reform Subcommittee A': 'Joint Tax Reform',
+    }
+
     def scrape(self, chamber, term):
         getattr(self, 'scrape_' + chamber)()
 
@@ -63,7 +71,16 @@ def scrape_interim_committee(self, link, name):
         doc = lxml.html.fromstring(html)
         doc.make_links_absolute(url)
 
-        comm = Committee('joint', name)
+        if 'Subcommittee' in name:
+            # Check whether the parent committee is manually defined first
+            # before attempting to automatically resolve it.
+            parent = WVCommitteeScraper.subcommittee_parent_map.get(name, None)
+            if parent is None:
+                parent = name.partition('Subcommittee')[0].strip()
+
+            comm = Committee('joint', parent, subcommittee=name)
+        else:
+            comm = Committee('joint', name)
         comm.add_source(url)
 
         xpath = '//a[contains(@href, "?member=")]'


Test patch:
Repository: openstates/openstates-scrapers

Issue:
NC - Committee assignments missing for a house member
State: NC

There may be more, I just happen to find this today. Jay Adams is missing committee assignments. The API does not show any in roles. 

https://openstates.org/api/v1/legislators/NCL000338/

The NCGA site shows committee assignments here: http://www.ncleg.net/gascripts/members/reports/committeeAssignments.pl?nUserid=697&Chamber=H

There may be something broken and it may affect more members in NC.

I have assignments for others. Have not determined what is different between Adams and the others


Discussion:
Also, just noticed the API is open now? I clicked the link and was able to see the data without a key.
thanks for this report, I'll look into it

(& API is indeed open at the moment, without Sunlight's old API key infrastructure we decided to wait and prioritize data quality)
Hey @jamesturk any luck with this?
Taking a look

Resolving patch:
diff --git a/openstates/nc/committees.py b/openstates/nc/committees.py
--- a/openstates/nc/committees.py
+++ b/openstates/nc/committees.py
@@ -17,10 +17,23 @@ def scrape_committee(self, committee, url):
             mtype, members = row.getchildren()
             if mtype.text == 'Members':
                 for m in members.getchildren():
-                    committee.add_member(m.text)
+                    member_name = self._clean_member_name(m.text)
+                    committee.add_member(member_name)
             else:
-                committee.add_member(members.text_content(), mtype.text)
+                member_name = self._clean_member_name(members.text_content())
+                committee.add_member(member_name, mtype.text)
 
+    def _clean_member_name(self, name):
+        """Names are displayed as "Office. LastName", e.g. "Rep. Adamsa". This strips the "Office. "
+
+        This helps the NameMatcher in billy link this to the correct legislator.
+        """
+        for prefix in ['Rep. ', 'Sen. ']:
+            if name.startswith(prefix):
+                return name.replace(prefix, '')
+
+        # If none hit, return the name as is
+        return name
 
     def scrape(self, term, chambers):
         base_url = 'http://www.ncga.state.nc.us/gascripts/Committees/Committees.asp?bPrintable=true&sAction=ViewCommitteeType&sActionDetails='


Test patch:
Repository: openstates/openstates-scrapers

Issue:
Link Docker's `data` (and maybe `cache`) directory to local file system
> @mshenfield [12:47 PM] I think we could make it easier by changing some of the configuration in `docker-compose` or the docker files so that it would write `data` to the openstates repository directory.

This is also something that @jamesturk and I have talked about internally. Let's get the `docker-compose` file updated so that users can see the `data` directory locally, and more easily hand-inspect the scraped JSON.

Maybe also link the `cache` directory, too, so that it's easier to throw out old scrapes? (But also fine if this is considered a bad idea, or if another solution to flushing old scrapes is suggested.)


Discussion:

Resolving patch:
diff --git a/billy_settings.py b/billy_settings.py
--- a/billy_settings.py
+++ b/billy_settings.py
@@ -1,24 +1,20 @@
 import os
 
-from os.path import abspath, dirname, join
-
 import sys
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 # This shims around folks that have openstates/ on virtualenv's .pth, but
 # not the root. This throws openstates.utils off, and isn't worth fiddling
 # that much with.
 
-SCRAPER_PATHS=[os.path.join(os.getcwd(), 'openstates')]
+SCRAPER_PATHS = [os.path.join(os.getcwd(), 'openstates')]
 MONGO_HOST = os.environ.get('BILLY_MONGO_HOST', 'localhost')
 MONGO_PORT = 27017
 MONGO_DATABASE = 'fiftystates'
 
-BILLY_MANUAL_DATA_DIR = os.environ.get('BILLY_MANUAL_DATA_DIR')
-if BILLY_MANUAL_DATA_DIR is None:
-    BILLY_MANUAL_DATA_DIR = os.path.join(
-        os.path.abspath(os.path.dirname(__file__)),
-        "manual_data",
-    )
+here = os.path.abspath(os.path.dirname(__file__))
+BILLY_DATA_DIR = os.environ.get('BILLY_DATA_DIR', os.path.join(here, 'data'))
+BILLY_CACHE_DIR = os.environ.get('BILLY_CACHE_DIR', os.path.join(here, 'cache'))
+BILLY_MANUAL_DATA_DIR = os.environ.get('BILLY_MANUAL_DATA_DIR', os.path.join(here, 'manual_data'))
 
 LEGISLATOR_FILTERS = {
     "billy.importers.filters.single_space_filter": [


Test patch:
Repository: openstates/openstates-scrapers

Issue:
CO: No legislation updates since Feb. 6
State: CO  (be sure to include in ticket title)

Hi, according to the API and the site ( https://openstates.org/co/ ), there haven't been updates to the Colorado legislation data since Feb. 6.


Discussion:

Resolving patch:
diff --git a/openstates/co/actions.py b/openstates/co/actions.py
--- a/openstates/co/actions.py
+++ b/openstates/co/actions.py
@@ -71,8 +71,11 @@
     Rule(u'(?i)refer (un)?amended to (?P<committees>.+)',
          [u'committee:referred']),
     Rule(u'(?i)\S+ Committee on (?P<committees>.+?) Refer (un)amended'),
+    Rule(u'Assigned to (<?P<committees>.+?)', 'committee:referred'),
     Rule(u'Second Reading Passed', [u'bill:reading:2']),
-    Rule(u'Third Reading Passed', ['bill:reading:3', 'bill:passed'])
+    Rule(u'Third Reading Passed', ['bill:reading:3', 'bill:passed']),
+    Rule(u'to Senate Committee of the Whole', 'committee:passed', actor='upper'),
+    Rule(u'to House Committee of the Whole', 'committee:passed', actor='lower')
     )
 
 committees_rgx = '(%s)' % '|'.join(
diff --git a/openstates/co/bills.py b/openstates/co/bills.py
--- a/openstates/co/bills.py
+++ b/openstates/co/bills.py
@@ -110,6 +110,7 @@ def scrape_bill(self, session, chamber, bill_url):
         self.scrape_versions(bill, page)
         self.scrape_research_notes(bill, page)
         self.scrape_fiscal_notes(bill, page)
+        self.scrape_committee_report(bill, page)
         self.scrape_votes(bill, page)
 
         self.save_bill(bill)
@@ -195,6 +196,12 @@ def scrape_research_notes(self, bill, page):
             note_url = note[0]
             bill.add_document("Research Note", note_url, 'application/pdf')
 
+    def scrape_committee_report(self, bill, page):
+        note = page.xpath('//a[text()="Committee Report"]/@href')
+        if note:
+            note_url = note[0]
+            bill.add_version("Committee Amendment", note_url, 'application/pdf')
+
     def scrape_votes(self, bill, page):
         votes = page.xpath('//div[@id="bill-documents-tabs3"]//table//tbody//tr')
 
@@ -217,7 +224,8 @@ def scrape_votes(self, bill, page):
             elif 'House' in header.group('committee'):
                 chamber = 'lower'
             else:
-                self.warning("No committee for %s" % header.group('committee'))
+                self.warning("No chamber for %s" % header.group('committee'))
+                chamber = bill['chamber']
 
             date = dt.datetime.strptime(header.group('date'), '%m/%d/%Y')
 


Test patch:
Repository: openstates/openstates-scrapers

Issue:
SD: legislators have no email addresses
Example legislator: http://sdlegislature.gov/Legislators/Legislators/MemberDetail.aspx?Session=2017&Member=1125&Cleaned=True

Looking at him in openstates, his phone number is there but his email address is missing: https://openstates.org/api/v1/legislators/SDL000286/

The data is there, the scraper just needs to be updated to capture the email property.

Thanks!


Discussion:

Resolving patch:
diff --git a/openstates/sd/legislators.py b/openstates/sd/legislators.py
--- a/openstates/sd/legislators.py
+++ b/openstates/sd/legislators.py
@@ -69,12 +69,14 @@ def scrape_legislator(self, name, chamber, term, url):
         if office_phone.strip() != "":
             kwargs['phone'] = office_phone
 
-        if email and email.strip() != "":
-            # South Dakota protects their email addresses from scraping using
-            # some JS code that runs on page load
-            # Until that code is run, all their email addresses are listed as
-            # *@example.com; so, fix this
-            kwargs['email'] = re.sub(r'@example\.com$', '@sdlegislature.gov', email)
+        # SD is hiding their email addresses entirely in JS now, so
+        # search through <script> blocks looking for them
+        for script in page.xpath('//script'):
+            if script.text:
+                match = re.search(r'([\w.]+@sdlegislature\.gov)', script.text)
+                if match:
+                    kwargs['email'] = match.group(0)
+                    break
 
         if kwargs:
             legislator.add_office('capitol', 'Capitol Office', **kwargs)


Test patch:
Repository: openstates/openstates-scrapers

Issue:
RI: state legislator first names contain middle initials
For example, https://openstates.org/api/v1/legislators/RIL000179/ has first_name of "Moira J." and nothing in middle_name for Moira J. Walsh.


Discussion:

Resolving patch:
diff --git a/openstates/ri/legislators.py b/openstates/ri/legislators.py
--- a/openstates/ri/legislators.py
+++ b/openstates/ri/legislators.py
@@ -128,8 +128,14 @@ def scrape(self, chamber, term):
             if d['address'] is '':
                 d['address'] = 'No Address Found'
 
+            # RI is very fond of First M. Last name formats and
+            # they're being misparsed upstream, so fix here
+            (first, middle, last) = ('','','')
+            if re.match(r'^\S+\s[A-Z]\.\s\S+$', full_name):
+                (first, middle, last) = full_name.split()
+                
             leg = Legislator(term, chamber, district_name, full_name,
-                             '', '', '',
+                             first, last, middle,
                              translate[d['party']],
                              **kwargs)
 


Test patch:
Repository: openstates/openstates-scrapers

Issue:
MA: Where are the Republicans?
All state legislators in MA are in either the Democratic party or Other.  It's a nice fantasy but it's not reality!



Discussion:

Resolving patch:
diff --git a/openstates/ma/legislators.py b/openstates/ma/legislators.py
--- a/openstates/ma/legislators.py
+++ b/openstates/ma/legislators.py
@@ -78,9 +78,9 @@ def scrape_member(self, chamber, term, member_url):
         party = party.strip()
         district = clean_district(district.strip())
 
-        if party == 'Democrat':
+        if party in ('D', 'Democrat', 'Democratic'):
             party = 'Democratic'
-        elif party == 'R':
+        elif party in ('R', 'Republican'):
             party = 'Republican'
         else:
             party = 'Other'


Test patch:
Repository: openstates/openstates-scrapers

Issue:
MA: committee scraper for 2017
State: MA

says that it is skipping every page, I believe the site was rewritten and so will need a complete rewrite


Discussion:

Resolving patch:
diff --git a/openstates/ma/committees.py b/openstates/ma/committees.py
--- a/openstates/ma/committees.py
+++ b/openstates/ma/committees.py
@@ -16,8 +16,6 @@ def scrape(self, term, chambers):
                            'House': 'lower',
                            'Joint': 'joint'}
 
-        foundComms = []
-
         for page_type in page_types:
             url = 'http://www.malegislature.gov/Committees/' + page_type
 
@@ -33,27 +31,15 @@ def scrape_committee(self, chamber, url):
         html = self.get(url, verify=False).text
         doc = lxml.html.fromstring(html)
 
-        name = doc.xpath('//span[@class="committeeShortName"]/text()')
-        if len(name) == 0:
-            self.warning("Had to skip this malformed page.")
-            return
-        # Because of http://www.malegislature.gov/Committees/Senate/S29 this
-        # XXX: hack had to be pushed in. Remove me ASAP. This just skips
-        #      malformed pages.
-
-        name = name[0]
+        name = doc.xpath('//title/text()')[0]
         com = Committee(chamber, name)
         com.add_source(url)
 
-        # get both titles and names, order is consistent
-        titles = doc.xpath('//p[@class="rankingMemberTitle"]/text()')
-        names = doc.xpath('//p[@class="rankingMemberName"]/a/text()')
-
-        for title, name in zip(titles, names):
-            com.add_member(name, title)
-
-        for member in doc.xpath('//div[@class="committeeRegularMembers"]//a/text()'):
-            com.add_member(member)
+        members = doc.xpath('//a[contains(@href, "/Legislators/Profile")]')
+        for member in members:
+            title = member.xpath('../span')
+            role = title[0].text.lower() if title else 'member'
+            com.add_member(member.text, role)
 
         if com['members']:
             self.save_committee(com)


Test patch:
Repository: openstates/openstates-scrapers

Issue:
AR scraper failing since at least 2017-03-11
State: AR - scraper has been failing since 2017-03-11

Based on automated runs it appears that AR has not run successfully in 2 days (2017-03-11).

```/usr/local/bin/billy-update ar``` | **failed during bills**

```
  Traceback (most recent call last):
  File "/opt/openstates/billy/billy/bin/update.py", line 368, in main
    run_record += _run_scraper(stype, args, metadata)
  File "/opt/openstates/billy/billy/bin/update.py", line 102, in _run_scraper
    scraper.scrape(chamber, time)
  File "/srv/openstates-web/openstates/ar/bills.py", line 40, in scrape
    self.save_bill(bill)
  File "/opt/openstates/billy/billy/scrape/__init__.py", line 199, in save_object
    self.validate_json(obj)
  File "/opt/openstates/billy/billy/scrape/__init__.py", line 130, in validate_json
    raise ve
FieldValidationError: Value u'' for field '<obj>.sponsors[0].name' cannot be blank'

```

Visit http://bobsled.openstates.org/ for more info.



Discussion:
@jamesturk, I would like to work on this issue, please help me get started, where to start for bills scraper testing and writing.

Resolving patch:
diff --git a/openstates/ar/bills.py b/openstates/ar/bills.py
--- a/openstates/ar/bills.py
+++ b/openstates/ar/bills.py
@@ -69,7 +69,8 @@ def scrape_bill(self, chamber, session):
             primary = row[11]
             if not primary:
                 primary = row[12]
-            bill.add_sponsor('primary', primary)
+            if primary:
+                bill.add_sponsor('primary', primary)
 
             # ftp://www.arkleg.state.ar.us/Bills/
             # TODO: Keep on eye on this post 2017 to see if they apply R going forward.


Test patch:
Repository: openstates/openstates-scrapers

Issue:
LA: "Total --" being saved as vote name
The [LA bill scraper](https://github.com/openstates/openstates/blame/master/openstates/la/bills.py#L180) is saving vote names as "Total --" instead of actual names.

Some examples:
Total -- 35, Total -- 0, Total -- 4, Total -- 34, Total -- 5

This is causing the vote to come back with a null leg_id, since the name cannot be matched back to a known legislator.


Discussion:

Resolving patch:
diff --git a/openstates/la/bills.py b/openstates/la/bills.py
--- a/openstates/la/bills.py
+++ b/openstates/la/bills.py
@@ -150,7 +150,6 @@ def scrape_vote(self, bill, name, url):
         os.remove(temp_path)
 
         vote_type = None
-        total_re = re.compile('^Total--(\d+)$')
         body = html.xpath('string(/html/body)')
 
         date_match = re.search('Date: (\d{1,2}/\d{1,2}/\d{4})', body)
@@ -164,7 +163,8 @@ def scrape_vote(self, bill, name, url):
 
         for line in body.replace(u'\xa0', '\n').split('\n'):
             line = line.replace('&nbsp;', '').strip()
-            if not line:
+            # Skip blank lines and "Total --"
+            if not line or 'Total --' in line:
                 continue
 
             if line in ('YEAS', 'NAYS', 'ABSENT'):
@@ -173,10 +173,7 @@ def scrape_vote(self, bill, name, url):
             elif line in ('Total', '--'):
                 vote_type = None
             elif vote_type:
-                match = total_re.match(line)
-                if match:
-                    vote['%s_count' % vote_type] = int(match.group(1))
-                elif vote_type == 'yes':
+                if vote_type == 'yes':
                     vote.yes(line)
                 elif vote_type == 'no':
                     vote.no(line)


Test patch:
Repository: openstates/openstates-scrapers

Issue:
MD: Vote names saved with a trailling "*"
The [MD bill scraper](http://github.com/openstates/openstates/blob/97eb4dbb147ef917b18db3896b1dc59ee62e71e9/openstates/md/bills.py#L333) saves vote names with a trailing "*".

Some examples:
Fisher*, Saab*, Glass*, Malone*, Kittleman*, Metzgar*, Atterbeary*, Ebersole*, Pendergrass*, Reilly*, McMillan*, Gutierrez*, Frush*, Mautz*

This is causing the vote to come back with a null leg_id, since the name cannot be matched back to a known legislator.


Discussion:
Here are [two](http://mgaleg.maryland.gov/webmga/frmMain.aspx?pid=flrvotepage&tab=subject3&id=hb0004,h-0496&stab=02&ys=2017RS) [examples](http://mgaleg.maryland.gov/webmga/frmMain.aspx?pid=flrvotepage&tab=subject3&id=hb0001%2ch-0212&stab=02&ys=2017RS) of the trailing "*".   I don't know what it indicates, but I think it is safe to strip out.


Resolving patch:
diff --git a/openstates/md/bills.py b/openstates/md/bills.py
--- a/openstates/md/bills.py
+++ b/openstates/md/bills.py
@@ -218,6 +218,7 @@ def parse_vote_page(self, vote_url):
             elif td.startswith('Excused'):
                 func = vote.other
             elif func:
+                td = td.rstrip('*')
                 func(td)
 
         return vote
@@ -302,7 +303,7 @@ def scrape_vote(self, bill, action_text, url):
             else:
                 passed = None
         elif 'overridden' in motion:
-            passed = True 
+            passed = True
             motion = 'Veto Override'
         else:
             raise Exception('unknown motion: %s' % motion)
@@ -332,11 +333,13 @@ def scrape_vote(self, bill, action_text, url):
                 vfunc = vote.other
             elif vfunc:
                 if ' and ' in text:
-                    a, b = text.split(' and ')
-                    vfunc(a)
-                    vfunc(b)
+                    legs = text.split(' and ')
                 else:
-                    vfunc(text)
+                    legs = [text]
+                for leg in legs:
+                    # Strip the occasional asterisk - see #1512
+                    leg = leg.rstrip('*')
+                    vfunc(leg)
 
         vote.validate()
         vote.add_source(url)


Test patch: