internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: openstates/openstates-scrapers

Issue:
MS failing since at least 2017-12-31
MS has been failing since 2017-12-31

Based on automated runs it appears that MS has not run successfully in 7 days (2017-12-31).


```
  00:00:18 CRITICAL pupa: Session(s) 2018 Regular Session were reported by Mississippi.get_session_list() but were not found in Mississippi.legislative_sessions or Mississippi.ignored_scraped_sessions.
no pupa_settings on path, using defaults
ms (scrape, import)
  bills: {}
  committee: {}
  people: {}
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/ms/people.py b/openstates/ms/people.py
--- a/openstates/ms/people.py
+++ b/openstates/ms/people.py
@@ -99,14 +99,12 @@ def scrape_details(self, chamber, leg_name, leg_link, role):
             email_name = root.xpath('string(//EMAIL_ADDRESS)').strip()
             cap_room = root.xpath('string(//CAP_ROOM)')
 
-            if leg_name in ('Lataisha Jackson', 'John G. Faulkner', 'Cheikh Taylor'):
+            if leg_name in ('Lataisha Jackson', 'John G. Faulkner'):
                 assert not party, ("Remove special-casing for this Democrat without a "
                                    "listed party: {}").format(leg_name)
                 party = 'Democratic'
             elif leg_name in ('James W. Mathis',
-                              'John Glen Corley',
-                              'Missy McGee',
-                              'Kevin Ford'):
+                              'John Glen Corley'):
                 assert not party, ("Remove special-casing for this Republican without"
                                    " a listed party: {}").format(leg_name)
                 party = 'Republican'


Test patch:
Repository: openstates/openstates-scrapers

Issue:
NH: 2018 session is not being scraped.
PR forthcoming.


Discussion:

Resolving patch:
diff --git a/billy_metadata/nh.py b/billy_metadata/nh.py
--- a/billy_metadata/nh.py
+++ b/billy_metadata/nh.py
@@ -16,7 +16,7 @@
          'start_year': 2013, 'end_year': 2014},
         {'name': '2015-2016', 'sessions': ['2015', '2016'],
          'start_year': 2015, 'end_year': 2016},
-        {'name': '2017-2018', 'sessions': ['2017'],
+        {'name': '2017-2018', 'sessions': ['2017', '2018'],
          'start_year': 2017, 'end_year': 2018}
     ],
     'session_details': {
@@ -55,6 +55,9 @@
         '2017': {'display_name': '2017 Regular Session',
                  '_scraped_name': '2017 Session',
                  },
+        '2018': {'display_name': '2018 Regular Session',
+                 '_scraped_name': '2018 Session',
+                 },
     },
     'feature_flags': ['subjects', 'influenceexplorer'],
     '_ignored_scraped_sessions': ['2013 Session', '2017 Session Bill Status Tables Link.txt'],
diff --git a/openstates/nh/__init__.py b/openstates/nh/__init__.py
--- a/openstates/nh/__init__.py
+++ b/openstates/nh/__init__.py
@@ -58,7 +58,14 @@ class NewHampshire(Jurisdiction):
             "name": "2017 Regular Session",
             "start_date": "2017-01-04",
             "end_date": "2017-06-30",
-        }
+        },
+        {
+            "_scraped_name": "2018 Session",
+            "end_date": "2018-06-30",
+            "identifier": "2018",
+            "name": "2018 Regular Session",
+            "start_date": "2018-01-03",
+        },
     ]
     ignored_scraped_sessions = [
         "2013 Session",
diff --git a/openstates/nh/bills.py b/openstates/nh/bills.py
--- a/openstates/nh/bills.py
+++ b/openstates/nh/bills.py
@@ -15,7 +15,9 @@
     'CR': 'concurrent resolution',
     'JR': 'joint resolution',
     'CO': 'concurrent order',
-    'A': 'address',
+    # really "bill of address";
+    # see https://github.com/opencivicdata/python-opencivicdata/issues/115
+    'A': 'bill',
 }
 action_classifiers = [
     ('Minority Committee Report', None),        # avoid calling these passage


Test patch:
Repository: openstates/openstates-scrapers

Issue:
OH: `get_session_list` cannot "see" current session
Ohio's `get_session_list` appears to have the current session _manually appended_. Not to mention, it hasn't been updated in a year. This should be fixed, to automatically "guess" the current session instead.

https://github.com/openstates/openstates/blob/master/openstates/oh/__init__.py#L91-L92


Discussion:
I am working on it.
Do we just want the current session because the code(https://github.com/openstates/openstates/blob/master/openstates/oh/__init__.py#L88) just scrapes the sessions till 130. And after that two more sessions(131, 132) are there. If we are extracting only 132 which is the current session, should we leave 131 or should we scrape both. There is one more way if the current session is 132, obviously 131 would also be there. For this we can append sessions which are there in between the current session and session 130.

Resolving patch:
diff --git a/openstates/oh/__init__.py b/openstates/oh/__init__.py
--- a/openstates/oh/__init__.py
+++ b/openstates/oh/__init__.py
@@ -46,14 +46,7 @@ class Ohio(Jurisdiction):
             "end_date": "2017-12-31"
         }
     ]
-    ignored_scraped_sessions = [
-        "127",
-        "126",
-        "125",
-        "124",
-        "123",
-        "122"
-    ]
+    ignored_scraped_sessions = []
 
     def get_organizations(self):
         legislature_name = "Ohio General Assembly"
@@ -85,9 +78,7 @@ def get_organizations(self):
         yield lower
 
     def get_session_list(self):
-        sessions = url_xpath('http://archives.legislature.state.oh.us',
-                             '//form[@action="bill_search.cfm"]//input[@type="radio"'
-                             ' and @name="SESSION"]/@value')
+        sessions = url_xpath('https://www.legislature.ohio.gov/legislation/search-legislation',
+                             '//div[@class="selectedValues"]/ul/span/li/text()')
         # Archive does not include current session
-        sessions.append('131')
         return sessions


Test patch:
Repository: openstates/openstates-scrapers

Issue:
OK failing since at least 2018-05-06
OK has been failing since 2018-05-06

Based on automated runs it appears that OK has not run successfully in 2 days (2018-05-06).


```
  /opt/openstates/venv-pupa/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
01:03:53 CRITICAL pupa: Session(s) 2019 Regular Session were reported by Oklahoma.get_session_list() but were not found in Oklahoma.legislative_sessions or Oklahoma.ignored_scraped_sessions.
loaded Open States pupa settings...
ok (scrape, import)
  bills: {}
  people: {}
  committees: {}
```

Visit http://bobsled.openstates.org for more info.



Discussion:
I'll get this.

Resolving patch:
diff --git a/openstates/ok/__init__.py b/openstates/ok/__init__.py
--- a/openstates/ok/__init__.py
+++ b/openstates/ok/__init__.py
@@ -69,6 +69,7 @@ class Oklahoma(Jurisdiction):
         },
     ]
     ignored_scraped_sessions = [
+        "2019 Regular Session",
         "2017 Regular Session",
         "2015 Regular Session",
         "2013 Regular Session",


Test patch:
Repository: openstates/openstates-scrapers

Issue:
ND failing since at least 2018-05-05
ND has been failing since 2018-05-05

Based on automated runs it appears that ND has not run successfully in 2 days (2018-05-05).


```
  /opt/openstates/venv-pupa/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
02:02:07 INFO scrapelib: GET - http://www.legis.nd.gov/assembly/
02:02:07 CRITICAL pupa: Session(s) 1862 1st Territorial Assembly, 1868-69 Territorial Assembly, 1885 Territorial Assembly, 1877 Territorial Assembly, 1872-73 Territorial Assembly, 1887 Territorial Assembly, 1881 Territorial Assembly, 1889 Territorial Assembly, 1862 2nd Territorial Assembly, 1874-75 Territorial Assembly, 1870-71 Territorial Assembly, 1864-65 Territorial Assembly, 1866-67 Territorial Assembly, 1883 Territorial Assembly, 1867-68 Territorial Assembly, 1879 Territorial Assembly, 1865-66 Territorial Assembly, 1863-64 Territorial Assembly were reported by NorthDakota.get_session_list() but were not found in NorthDakota.legislative_sessions or NorthDakota.ignored_scraped_sessions.
loaded Open States pupa settings...
nd (scrape, import)
  people: {}
  votes: {}
  bills: {}
  committees: {}
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/nd/__init__.py b/openstates/nd/__init__.py
--- a/openstates/nd/__init__.py
+++ b/openstates/nd/__init__.py
@@ -146,4 +146,6 @@ def get_session_list(self):
         html = scrapelib.Scraper().get(url).text
         doc = lxml.html.fromstring(html)
         doc.make_links_absolute(url)
-        return doc.xpath("//div[@class='view-content']//a/text()")
+        sessions = doc.xpath("//div[@class='view-content']//a/text()")
+        sessions = [session for session in sessions if 'Territorial Assembly' not in session]
+        return sessions


Test patch:
Repository: openstates/openstates-scrapers

Issue:
InsecureRequestWarning log spam
Scrape logs for https sites are spammed with this INFO-level message on every HTTPS request:
```
/opt/openstates/venv-pupa/lib/python3.5/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
```

I'm looking for advice about what should be done.  My inclination is to quell the warnings altogether, because I _suspect_ that stale state certs are frequent enough to not want to bother with verification.  I believe (but have not tested) that this can be done in openstates with

```py
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
```

If we want to  verify certs, it probably requires changes somewhere up the stack.



Discussion:

Resolving patch:
diff --git a/openstates/__init__.py b/openstates/__init__.py
--- a/openstates/__init__.py
+++ b/openstates/__init__.py
@@ -0,0 +1,4 @@
+import urllib3
+
+# Quell InsecureRequestWarning: Unverified HTTPS request warnings
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


Test patch:
Repository: openstates/openstates-scrapers

Issue:
CT House Votes Missing
State: CT

No assembly votes appear to be pulled in for the current regular session.

**Addendum 1:** Senate votes are also affected.

**Addendum 2:** It seems in the `scrape_votes()` function, when [parsing the HTML table](https://github.com/openstates/openstates/blob/master/openstates/ct/bills.py#L187), the `xpath` logic is off. It scrapes the "Y", "N", "X" as the legislator name (it did this for [Vote for HB-7052 Roll Call Number 279](https://www.cga.ct.gov/2017/VOTE/h/2017HV-00279-R00HB07052-HV.htm), for example), and treats them all as "other" votes...


Discussion:
On VoteEvent Details, Page votes count other than "yes' and "no" are displayed as `Those absent and not voting` and only sign "X" is used in the table (eg, [Link](https://www.cga.ct.gov/2017/VOTE/h/2017HV-00279-R00HB07052-HV.htm)) 
and in OCD `absent` and `not voting` are two vote option choices and here we can't differ between in these two choices. 
so I think it's ok for now to leave them as "other" votes.
Looking on this one.

Resolving patch:
diff --git a/openstates/ct/bills.py b/openstates/ct/bills.py
--- a/openstates/ct/bills.py
+++ b/openstates/ct/bills.py
@@ -126,7 +126,7 @@ def scrape_bill_page(self, bill):
                                             link.attrib['href'])
 
     def scrape_vote(self, bill, name, url):
-        if "VOTE/H" in url:
+        if "VOTE/h" in url:
             vote_chamber = 'lower'
             cols = (1, 5, 9, 13)
             name_offset = 3
@@ -183,7 +183,6 @@ def scrape_vote(self, bill, name, url):
         vote.set_count('no', no_count)
         vote.set_count('other', other_count)
         vote.add_source(url)
-
         table = page.xpath("//table")[0]
         for row in table.xpath("tr"):
             for i in cols:


Test patch:
Repository: openstates/openstates-scrapers

Issue:
LA failing since at least 2018-06-17
LA has been failing since 2018-06-17

Based on automated runs it appears that LA has not run successfully in 2 days (2018-06-17).


```
  /opt/openstates/venv-pupa/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
01:01:37 CRITICAL pupa: Session(s) 2018 Third Extraordinary Session were reported by Louisiana.get_session_list() but were not found in Louisiana.legislative_sessions or Louisiana.ignored_scraped_sessions.
loaded Open States pupa settings...
la (scrape, import)
  people: {}
  committees: {}
  bills: {}
```

Visit http://bobsled.openstates.org for more info.



Discussion:
I'm on this one.

Resolving patch:
diff --git a/billy_metadata/la.py b/billy_metadata/la.py
--- a/billy_metadata/la.py
+++ b/billy_metadata/la.py
@@ -48,6 +48,7 @@
                 '2018 1st Extraordinary Session',
                 '2018',
                 '2018 2nd Extraordinary Session',
+                '2018 3rd Extraordinary Session',
             ],
         },
     ],
@@ -172,6 +173,10 @@
             'type': 'special',
             'display_name': '2018, 2nd Extraordinary Session',
         },
+        "2018 3rd Extraordinary Session": {
+            'type': 'special',
+            'display_name': '2018, 3rd Extraordinary Session',
+        },
     },
     'feature_flags': ['subjects', 'influenceexplorer', 'events'],
     '_ignored_scraped_sessions': [
diff --git a/openstates/la/__init__.py b/openstates/la/__init__.py
--- a/openstates/la/__init__.py
+++ b/openstates/la/__init__.py
@@ -153,6 +153,13 @@ class Louisiana(Jurisdiction):
             "name": "2018, 2nd Extraordinary Session",
             "start_date": "2018-05-22",
         },
+        {
+            "_scraped_name": "2018 Third Extraordinary Session",
+            "classification": "special",
+            "identifier": "2018 3rd Extraordinary Session",
+            "name": "2018, 3rd Extraordinary Session",
+            "start_date": "2018-06-19",
+        },
     ]
     ignored_scraped_sessions = [
         "2016 Organizational Session",
diff --git a/openstates/la/bills.py b/openstates/la/bills.py
--- a/openstates/la/bills.py
+++ b/openstates/la/bills.py
@@ -31,6 +31,7 @@ class LABillScraper(Scraper, LXMLMixin):
         '2018 1st Extraordinary Session': '181ES',
         '2018': '18RS',
         '2018 2nd Extraordinary Session': '182ES',
+        '2018 3rd Extraordinary Session': '183ES',
     }
 
     def pdf_to_lxml(self, filename, type='html'):


Test patch:
Repository: openstates/openstates-scrapers

Issue:
MN failing since at least 2018-06-15
MN has been failing since 2018-06-15

Based on automated runs it appears that MN has not run successfully in 2 days (2018-06-15).


```
  /opt/openstates/venv-pupa/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
02:01:53 CRITICAL pupa: Session(s) 91st Legislature, 2019-2020 were reported by Minnesota.get_session_list() but were not found in Minnesota.legislative_sessions or Minnesota.ignored_scraped_sessions.
loaded Open States pupa settings...
mn (scrape, import)
  bills: {}
  committees: {}
  people: {}
  vote_events: {}
```

Visit http://bobsled.openstates.org for more info.



Discussion:

Resolving patch:
diff --git a/openstates/mn/__init__.py b/openstates/mn/__init__.py
--- a/openstates/mn/__init__.py
+++ b/openstates/mn/__init__.py
@@ -127,6 +127,7 @@ class Minnesota(Jurisdiction):
         '79th Legislature, 1995-1996',
         '79th Legislature, 1995 1st Special Session',
         '89th Legislature, 2015-2016',
+        '91st Legislature, 2019-2020',
     ]
 
     def get_organizations(self):


Test patch:
Repository: openstates/openstates-scrapers

Issue:
PA failing since at least 2018-06-20
PA has been failing since 2018-06-20

Based on automated runs it appears that PA has not run successfully in 2 days (2018-06-20).


```
  loaded Open States pupa settings...
pa (scrape, import)
/opt/openstates/venv-pupa/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  bills: {}
  people: {}
  committees: {}
  """)
00:02:17 CRITICAL pupa: Session(s) 1962 Regular Session were reported by Pennsylvania.get_session_list() but were not found in Pennsylvania.legislative_sessions or Pennsylvania.ignored_scraped_sessions.
```

Visit http://bobsled.openstates.org for more info.



Discussion:
I'll grab this

Resolving patch:
diff --git a/openstates/pa/__init__.py b/openstates/pa/__init__.py
--- a/openstates/pa/__init__.py
+++ b/openstates/pa/__init__.py
@@ -224,5 +224,10 @@ def get_organizations(self):
         yield lower
 
     def get_session_list(self):
+        # PA keeps slowly adding backdata, so just ignore it en masse
+        for i in range(1800, 2000):
+            self.ignored_scraped_sessions.append('{} Regular Session'.format(i))
+            self.ignored_scraped_sessions.append('{} Special Session #1'.format(i))
+
         return url_xpath('http://www.legis.state.pa.us/cfdocs/legis/home/bills/',
                          '//select[@id="billSessions"]/option/text()')


Test patch:
Repository: openstates/openstates-scrapers

Issue:
VA: no senate votes
State: VA

no votes in VA senate right now, needs investigation


Discussion:
@jamesturk should I look into this?
There are votes now, but roll calls are either not being scraped or not resolved.
taking this.


Resolving patch:
diff --git a/openstates/va/bills.py b/openstates/va/bills.py
--- a/openstates/va/bills.py
+++ b/openstates/va/bills.py
@@ -238,6 +238,7 @@ def handle_page(self):
                     cached_vote.set_count('no', n)
                     cached_vote.set_count('other', o)
                     if vote_url:
+                        list(self.scrape_page_items(VotePage, url=vote_url[0], obj=cached_vote))
                         cached_vote.add_source(vote_url[0])
                     else:
                         cached_vote.add_source(self.url)
@@ -250,7 +251,6 @@ def handle_page(self):
                                 counts['no'] == n and
                                 counts['other'] == o):
                             vote = cached_vote
-                            list(self.scrape_page_items(VotePage, url=vote_url[0], obj=vote))
                             vote.add_source(vote_url[0])
                             action = cached_action
                     elif cached_vote.motion_text.startswith('VOTE:'):
@@ -278,6 +278,8 @@ def handle_page(self):
                         cached_vote.set_count('other', o)
                         if vote_url:
                             cached_vote.add_source(vote_url[0])
+                            list(self.scrape_page_items(
+                                VotePage, url=vote_url[0], obj=cached_vote))
                         else:
                             cached_vote.add_source(self.url)
                         cached_action = action


Test patch:
Repository: openstates/openstates-scrapers

Issue:
GA votes need name matching
The GA state senate is missing votes (e.g., [GAV00006507](https://openstates.org/ga/votes/GAV00006507/) doesn't have a single vote) **and** the state Assembly is missing votes (e.g., [GAV00006524](https://openstates.org/ga/votes/GAV00006524/) doesn't have a single vote). This appears quite widespread for the current term in GA, that votes seem to be missing on nearly every motion.

**Addendum.** The problem appears to be the clerks record votes as "LAST NAME, district number" which messes up the vote parsing. But if we simply remove the ", district number" part of the string, the votes are identified with the legislator correctly, which seems to fix the problem.


Discussion:
I believe this failure has become total; there are now no individual roll call votes for 2017.

Resolving patch:
diff --git a/openstates/ga/bills.py b/openstates/ga/bills.py
--- a/openstates/ga/bills.py
+++ b/openstates/ga/bills.py
@@ -1,3 +1,4 @@
+import re
 from collections import defaultdict
 
 from pupa.scrape import Scraper, Bill, VoteEvent
@@ -26,6 +27,8 @@
 member_cache = {}
 SOURCE_URL = 'http://www.legis.ga.gov/Legislation/en-US/display/{session}/{bid}'
 
+vote_name_pattern = re.compile(r'(.*), (\d+(?:ST|ND|RD|TH))', re.IGNORECASE)
+
 
 class GABillScraper(Scraper):
     lservice = get_client('Legislation').service
@@ -193,7 +196,10 @@ def scrape(self, session=None, chamber=None):
                     for vdetail in vote_['Votes'][0]:
                         whom = vdetail['Member']
                         how = vdetail['MemberVoted']
-                        vote.vote(methods.get(how, 'other'), whom['Name'])
+                        if whom['Name'] == 'VACANT':
+                            continue
+                        name, district = vote_name_pattern.search(whom['Name']).groups()
+                        vote.vote(methods.get(how, 'other'), name, note=district)
 
                     yield vote
 


Test patch: