internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: urllib3/urllib3

Issue:
Handle OpenSSL 3.2.0 new error message for unexpected unencrypted records
### Subject

I was expecting to all tests working on "main" branch but there are 3 failing test cases

* TestProxyManager::test_https_proxymanager_connected_to_http_proxy[http]
* TestProxyManager::test_https_proxymanager_connected_to_http_proxy[https]
* TestSSL::test_ssl_failure_midway_through_conn 

```
nox -s test-3.12 
...
FAILED test/with_dummyserver/test_socketlevel.py::TestProxyManager::test_https_proxymanager_connected_to_http_proxy[http] - assert 'Your proxy appears to only use HTTP and not HTTPS' in "('Unable to connect to proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1000)')))"
FAILED test/with_dummyserver/test_socketlevel.py::TestProxyManager::test_https_proxymanager_connected_to_http_proxy[https] - assert 'Your proxy appears to only use HTTP and not HTTPS' in "('Unable to connect to proxy', SSLError(SSLError(1, '[SSL] record layer failure (_ssl.c:1000)')))"
FAILED test/with_dummyserver/test_socketlevel.py::TestSSL::test_ssl_failure_midway_through_conn - AssertionError: Regex pattern did not match.
```

### Environment

```
OS macOS-14.2.1-x86_64-i386-64bit
Python 3.12.1
OpenSSL 3.2.0 23 Nov 2023
urllib3 2.1.0
```
### Steps to Reproduce

```
nox -s test-3.12 
```

### Expected Behavior

The test should work, or they should be skipped, or if it's because they are not supported on macOS, then may it should be conditionally skipped. 


### Actual Behavior

3 test fails 




Discussion:

Resolving patch:
diff --git a/src/urllib3/connection.py b/src/urllib3/connection.py
--- a/src/urllib3/connection.py
+++ b/src/urllib3/connection.py
@@ -862,6 +862,7 @@ def _wrap_proxy_error(err: Exception, proxy_scheme: str | None) -> ProxyError:
     is_likely_http_proxy = (
         "wrong version number" in error_normalized
         or "unknown protocol" in error_normalized
+        or "record layer failure" in error_normalized
     )
     http_proxy_warning = (
         ". Your proxy appears to only use HTTP and not HTTPS, "


Test patch:
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -1293,7 +1293,8 @@ def socket_handler(listener: socket.socket) -> None:
         self._start_server(socket_handler)
         with HTTPSConnectionPool(self.host, self.port, ca_certs=DEFAULT_CA) as pool:
             with pytest.raises(
-                SSLError, match=r"(wrong version number|record overflow)"
+                SSLError,
+                match=r"(wrong version number|record overflow|record layer failure)",
             ):
                 pool.request("GET", "/", retries=False)
Repository: urllib3/urllib3

Issue:
Move length_remaining into BaseHTTPResponse
Now that `HTTPResponse` and `HTTP2Response` both have a `length_remaining` attribute, it should be made part of `BaseHTTPResponse`. This will also remove one mypy ignore comment: https://github.com/urllib3/urllib3/pull/3311#pullrequestreview-1841977649.

I'm happy to provide more details if needed.


Discussion:
Hey @pquentin, see if the attached PR fixes this.

Resolving patch:
diff --git a/src/urllib3/connectionpool.py b/src/urllib3/connectionpool.py
--- a/src/urllib3/connectionpool.py
+++ b/src/urllib3/connectionpool.py
@@ -556,7 +556,7 @@ def _make_request(
             # HTTP version
             http_version,
             response.status,
-            response.length_remaining,  # type: ignore[attr-defined]
+            response.length_remaining,
         )
 
         return response
diff --git a/src/urllib3/response.py b/src/urllib3/response.py
--- a/src/urllib3/response.py
+++ b/src/urllib3/response.py
@@ -344,6 +344,7 @@ def __init__(
             self.chunked = True
 
         self._decoder: ContentDecoder | None = None
+        self.length_remaining: int | None
 
     def get_redirect_location(self) -> str | None | Literal[False]:
         """


Test patch:
Repository: urllib3/urllib3

Issue:
Emscripten support emits an InsecureRequestWarning even when using HTTPS
This is a side-effect of us using JavaScript APIs instead of Python TLS and setting `is_verified` on the `EmscriptenHTTPConnection` so urllib3 is emitting an `InsecureRequestWarning` for every request, even ones that are using HTTPS.

* Set the proper value of `is_verified` depending on whether the request is HTTP or HTTPS.
* Add a test case that asserts that an `InsecureRequestWarning` is emitted for HTTP and isn't emitted for HTTPS.



Discussion:

Resolving patch:
diff --git a/src/urllib3/contrib/emscripten/connection.py b/src/urllib3/contrib/emscripten/connection.py
--- a/src/urllib3/contrib/emscripten/connection.py
+++ b/src/urllib3/contrib/emscripten/connection.py
@@ -67,6 +67,7 @@ def __init__(
         self.blocksize = blocksize
         self.source_address = None
         self.socket_options = None
+        self.is_verified = False
 
     def set_tunnel(
         self,
@@ -228,6 +229,10 @@ def __init__(
 
         self.cert_reqs = None
 
+        # The browser will automatically verify all requests.
+        # We have no control over that setting.
+        self.is_verified = True
+
     def set_cert(
         self,
         key_file: str | None = None,


Test patch:
diff --git a/test/contrib/emscripten/test_emscripten.py b/test/contrib/emscripten/test_emscripten.py
--- a/test/contrib/emscripten/test_emscripten.py
+++ b/test/contrib/emscripten/test_emscripten.py
@@ -947,3 +947,28 @@ def count_calls(self, *args, **argv):  # type: ignore[no-untyped-def]
         assert count == 6
 
     pyodide_test(selenium_coverage, testserver_http.http_host, find_unused_port())
+
+
+@install_urllib3_wheel()
+def test_insecure_requests_warning(
+    selenium_coverage: typing.Any, testserver_http: PyodideServerInfo
+) -> None:
+    @run_in_pyodide  # type: ignore[misc]
+    def pyodide_test(selenium_coverage, host: str, port: int, https_port: int) -> None:  # type: ignore[no-untyped-def]
+        import warnings
+
+        import urllib3
+        import urllib3.exceptions
+
+        http = urllib3.PoolManager()
+
+        with warnings.catch_warnings(record=True) as w:
+            http.request("GET", f"https://{host}:{https_port}")
+        assert len(w) == 0
+
+    pyodide_test(
+        selenium_coverage,
+        testserver_http.http_host,
+        testserver_http.http_port,
+        testserver_http.https_port,
+    )
Repository: openstates/openstates-scrapers

Issue:
TX: version-less bills missing bill text(s)
No link to bill text for the version-less bills
examples:
http://openstates.org/browse/tx/82/HJR%20146/
http://www.legis.state.tx.us/BillLookup/Text.aspx?LegSess=82R&Bill=HJR146

http://openstates.org/browse/tx/81/HJR%20120/
http://www.legis.state.tx.us/BillLookup/Text.aspx?LegSess=81R&Bill=HJR120

http://openstates.org/browse/tx/81/HJR%2093/
http://www.legis.state.tx.us/BillLookup/Text.aspx?LegSess=81R&Bill=HJR93

http://openstates.org/browse/tx/82/HCR%2094/
http://www.legis.state.tx.us/BillLookup/Text.aspx?LegSess=82R&Bill=HCR94



Discussion:
Confirmed, thanks!


Resolving patch:
diff --git a/openstates/tx/bills.py b/openstates/tx/bills.py
--- a/openstates/tx/bills.py
+++ b/openstates/tx/bills.py
@@ -51,6 +51,7 @@ def scrape_bill(self, chamber, session, url):
             versions_url = '/'.join(versions_url.split('/')[0:-1])
 
             bill_prefix = bill['bill_id'].split()[0]
+            bill_prefix = bill_prefix.replace('JR', 'J').replace('CR', 'C')
             bill_num = int(bill['bill_id'].split()[1])
             long_bill_id = "%s%05d" % (bill_prefix, bill_num)
 


Test patch:
Repository: openstates/openstates-scrapers

Issue:
NY: Committees don't have members
Taken from bug #195

http://openstates.org/browse/ny/committees/
http://www.nysenate.gov/committee/state-native-american-relations
http://assembly.state.ny.us/comm/?sec=mem&id=60



Discussion:
Fixed by https://github.com/sunlightlabs/openstates/commit/4f6e3b927dc3f83ccc3e37c8441512870c6f7d8a

hmmm nope, for some reason NY still has 2 empty committees, but they shouldn't be

Do you have a link to which ones?

Senate Native American Relations:
http://www.nysenate.gov/committee/state-native-american-relations

Assembly Demographic Research...:
http://assembly.state.ny.us/comm/?sec=mem&id=60


Resolving patch:
diff --git a/openstates/ny/committees.py b/openstates/ny/committees.py
--- a/openstates/ny/committees.py
+++ b/openstates/ny/committees.py
@@ -68,7 +68,7 @@ def scrape_lower_committee(self, name, url):
         comm = Committee('lower', name)
         comm.add_source(url)
 
-        for link in page.xpath("//a[contains(@href, 'mem?ad')]"):
+        for link in page.xpath("//div[@class='commlinks']//a[contains(@href, 'mem')]"):
             member = link.text.strip()
             member = re.sub(r'\s+', ' ', member)
 


Test patch:
Repository: openstates/openstates-scrapers

Issue:
NY: Native American Relations committee is not being scraped correctly
http://www.nysenate.gov/committee/state-native-american-relations
http://openstates.org/admin/object_json/committees/NYC000116/



Discussion:
Forked from issue #247

Looks like scraping missing "Native American Relations" and "Libraries". They are both "Temporary Committees"

Are there any rules for what exactly you're trying to scrap? Include Temporary Committees, but skip Legislative Commissions?

Check the bottom of this page:
http://www.nysenate.gov/committees


Resolving patch:
diff --git a/openstates/ny/committees.py b/openstates/ny/committees.py
--- a/openstates/ny/committees.py
+++ b/openstates/ny/committees.py
@@ -84,15 +84,17 @@ def scrape_upper(self):
         page = lxml.html.fromstring(page)
         page.make_links_absolute(url)
 
-        for link in page.xpath("//a[contains(@href, '/committee/')]"):
-            name = link.text.strip()
+        for h2 in page.xpath("//h2"):
+            committee_types = ['Standing Committees','Temporary Committees']
+            if h2.text not in committee_types:
+                continue
+            
+            for link in h2.getparent().xpath(".//a[contains(@href, '/committee/')]"):
+                name = link.text.strip()
 
-            if name == 'New York State Conference of Black Senators':
-                # stop scraping once we reach the caucuses
-                break
+                committees.append(name)
+                self.scrape_upper_committee(name, link.attrib['href'])
 
-            committees.append(name)
-            self.scrape_upper_committee(name, link.attrib['href'])
         return committees
 
     def scrape_upper_committee(self, name, url):


Test patch:
Repository: openstates/openstates-scrapers

Issue:
TN: 109th Session, Legislators, Upper - Senator name extraction fails for district 19
Trying to run the scraper locally, I encountered the following exception for TN's [19th senate district in the 109th general assembly](http://www.capitol.tn.gov/senate/archives/109GA/members/s19.html) (Sen. Thelma Hale):

```
File "/usr/local/bin/billy-update", line 9, in <module>
    load_entry_point('billy==1.8.4', 'console_scripts', 'billy-update')()
  File "/opt/sunlightfoundation.com/billy/billy/bin/update.py", line 377, in main
    run_record += _run_scraper(stype, args, metadata)
  File "/opt/sunlightfoundation.com/billy/billy/bin/update.py", line 101, in _run_scraper
    scraper.scrape(chamber, time)
  File "/srv/openstates-web/openstates/tn/legislators.py", line 81, in scrape
    name = member_page.xpath('//div[@id="membertitle"]/h2/text()')[0]
IndexError: list index out of range
```


Discussion:

Resolving patch:
diff --git a/openstates/tn/legislators.py b/openstates/tn/legislators.py
--- a/openstates/tn/legislators.py
+++ b/openstates/tn/legislators.py
@@ -76,7 +76,7 @@ def scrape(self, chamber, term):
 
             member_page = lxml.html.fromstring(member_page)
             try:
-                name = member_page.xpath('body/div/div/h1/text()')[0]
+                name = member_page.xpath('//div/div/h1/text()')[0]
             except IndexError:
                 name = member_page.xpath('//div[@id="membertitle"]/h2/text()')[0]
             


Test patch:
Repository: openstates/openstates-scrapers

Issue:
Make `LXMLMixin.lxmlize` use scraper's `session`, rather than a raw `request.get()`
Otherwise it slows down `--fast` scrapes because it doesn't use cached responses.


Discussion:
I'm happy to take a look at this! I'll try and make a pr soon.

Resolving patch:
diff --git a/openstates/utils/lxmlize.py b/openstates/utils/lxmlize.py
--- a/openstates/utils/lxmlize.py
+++ b/openstates/utils/lxmlize.py
@@ -14,11 +14,13 @@ def lxmlize(self, url, raise_exceptions=False):
             Element: Document node representing the page.
         """
         try:
-            response = requests.get(url)
+            # This class is always mixed into subclasses of `billy.Scraper`,
+            # which have a `get` method defined.
+            response = self.get(url)
         except requests.exceptions.SSLError:
             self.warning('`self.lxmlize()` failed due to SSL error, trying'\
-                'an unverified `requests.get()`')
-            response = requests.get(url, verify=False)
+                'an unverified `self.get()` (i.e. `requests.get()`)')
+            response = self.get(url, verify=False)
 
         if raise_exceptions:
             response.raise_for_status()


Test patch:
Repository: openstates/openstates-scrapers

Issue:
MN: Bill sponsors wrong
MN is pulling "Separated" and "Chronological" as bill sponsors instead of the the correct names.
https://openstates.org/mn/bills/2017-2018/SF155/

Happy to PR to fix the xpath, but i'm unclear on who's a sponsor and who's a co-sponsor.

For instance, on:
https://www.revisor.mn.gov/bills/bill.php?b=House&f=HF0095&ssn=0&y=2017
Legiscan thinks only the first is the sponsor.
https://legiscan.com/MN/sponsors/HF95/2017

Can anyone confirm that the first is always the sponsor, and the rest are cosponsors?


Discussion:

Resolving patch:
diff --git a/openstates/mn/bills.py b/openstates/mn/bills.py
--- a/openstates/mn/bills.py
+++ b/openstates/mn/bills.py
@@ -347,17 +347,15 @@ def extract_sponsors(self, bill, doc, chamber):
         """
         Extracts sponsors from bill page.
         """
-        sponsors = doc.xpath('//h2[text()="Authors"]/following-sibling::ul[1]/li/a/text()')
-        if sponsors:
-            primary_sponsor = sponsors[0].strip()
-            bill.add_sponsor('primary', primary_sponsor, chamber=chamber)
-            cosponsors = sponsors[1:]
-            for leg in cosponsors:
-                bill.add_sponsor('cosponsor', leg.strip(), chamber=chamber)
-
-        other_sponsors = doc.xpath('//h3[contains(text(), "Authors")]/following-sibling::ul[1]/li/a/text()')
-        for leg in other_sponsors:
-            bill.add_sponsor('cosponsor', leg.strip(), chamber=self.other_chamber(chamber))
+        sponsors = doc.xpath('//div[@class="author"]/ul/li/a/text()')
+        for index, sponsor in enumerate(sponsors):
+            if index == 0:
+                sponsor_type = 'primary'
+            else:
+                sponsor_type = 'cosponsor'
+
+            sponsor_name = sponsor.strip()
+            bill.add_sponsor(sponsor_type, sponsor_name, chamber=chamber)
 
         return bill
 


Test patch:
Repository: openstates/openstates-scrapers

Issue:
NY: events aren't scraping
State: NY

Event scraper is broken & needs investigation.


Discussion:

Resolving patch:
diff --git a/openstates/ny/events.py b/openstates/ny/events.py
--- a/openstates/ny/events.py
+++ b/openstates/ny/events.py
@@ -26,12 +26,16 @@ def lower_parse_page(self, url, session):
                 if len(tds) < 2:
                     continue
                 key, value = tds
-                if key.tag == 'th':
+
+                if key.tag == 'th' and key.get("class") == 'hrgdate':
                     date = key.text_content()
-                    date = re.sub("\s+", " ", date)
+                    date = re.sub(r"\s+", " ", date)
                     date = re.sub(".*POSTPONED NEW DATE", "", date).strip()
-                    ctty = value.xpath(".//strong")[0]
-                    ctty = ctty.text_content()
+
+                # Due to the html structure this shouldn't be an elif
+                # It needs to fire twice in the same loop iteration
+                if value.tag == 'th' and value.get("class") == 'commtitle':
+                    ctty = value.text_content()
 
                     chamber = 'other'
                     if "senate" in ctty.lower():
@@ -40,11 +44,14 @@ def lower_parse_page(self, url, session):
                         chamber = 'lower'
                     if "joint" in ctty.lower():
                         chamber = 'joint'
+
+                    coms = value.xpath('.//div[contains(@class,"comm-txt")]/text()')
+
                 elif key.tag == 'td':
                     key = key.text_content().strip()
                     value = value.text_content().strip()
                     value = value.replace(u'\x96', '-')
-                    value = re.sub("\s+", " ", value)
+                    value = re.sub(r"\s+", " ", value)
                     metainf[key] = value
 
             time = metainf['Time:']
@@ -119,10 +126,12 @@ def lower_parse_page(self, url, session):
             if 'Media Contact:' in metainf:
                 event.update(media_contact=metainf['Media Contact:'])
             event.add_source(url)
-            event.add_participant('host',
-                                  ctty,
-                                  'committee',
-                                  chamber=chamber)
+
+            for com in coms:
+                event.add_participant('host',
+                                    com.strip(),
+                                    'committee',
+                                    chamber=self.classify_committee(com))
 
             self.save_event(event)
 
@@ -134,6 +143,16 @@ def scrape_lower(self, chamber, session):
         if chamber == 'other':
             self.lower_parse_page(url, session)
 
+    def classify_committee(self, name):
+        chamber = 'other'
+        if "senate" in name.lower():
+            chamber = 'upper'
+        if "assembly" in name.lower():
+            chamber = 'lower'
+        if "joint" in name.lower():
+            chamber = 'joint'
+        return chamber
+
     """
     def scrape_upper(self, chamber, session):
         if chamber != 'upper':


Test patch:
Repository: openstates/openstates-scrapers

Issue:
Louisiana is not yet available for 2017
State: LA

Louisiana's session starts on 4/10, we'll monitor the situation and update LA as soon as possible.



Discussion:
But they are starting to pre-file bills, e.g., HB6 for 2017 [is pre-filed and online](https://www.legis.la.gov/legis/BillInfo.aspx?i=231278&sbi=y).

Resolving patch:
diff --git a/openstates/la/__init__.py b/openstates/la/__init__.py
--- a/openstates/la/__init__.py
+++ b/openstates/la/__init__.py
@@ -49,6 +49,7 @@
                 '2016',
                 '2016 1st Extraordinary Session',
                 '2016 2nd Extraordinary Session',
+                '2017',
             ],
         },
     ],
@@ -141,6 +142,14 @@
             '_id': '162ES',
             '_scraped_name': '2016 Second Extraordinary Session',
         },
+        '2017': {
+            'type': 'primary',
+            'start_date': datetime.date(2017, 4, 10),
+            'end_date': datetime.date(2017, 6, 8),
+            'display_name': '2017 Regular Session',
+            '_id': '17RS',
+            '_scraped_name': '2017 Regular Session',
+        },
     },
     'feature_flags': ['subjects', 'influenceexplorer', 'events'],
     '_ignored_scraped_sessions': [


Test patch:
Repository: openstates/openstates-scrapers

Issue:
AZ Legislator with the following id has an invalid phone number AZL000372
State: AZ  (be sure to include in ticket title)

This repository is for issues with state data, for feature requests, etc.
please visit the contributor guide (see above message) to file the issue in the correct place.



Discussion:
the phone number is listed as  "phone": "602-602-926-4124"

Resolving patch:
diff --git a/openstates/az/legislators.py b/openstates/az/legislators.py
--- a/openstates/az/legislators.py
+++ b/openstates/az/legislators.py
@@ -1,8 +1,7 @@
-from billy.scrape import NoDataForPeriod
 from billy.scrape.legislators import LegislatorScraper, Legislator
 from lxml import html
+import re
 
-import re, datetime
 
 class AZLegislatorScraper(LegislatorScraper):
     jurisdiction = 'az'
@@ -80,30 +79,30 @@ def scrape(self, chamber, term):
                               + "\nPhoenix, AZ 85007"
 
             phone = phone.text_content().strip()
-            if not phone.startswith('602'):
+            if '602' not in re.findall(r'(\d+)', phone):
                 phone = "602-" + phone
 
             leg = Legislator(term, chamber, district, full_name=name,
-                                party=party, url=link,
-                                photo_url=photo_url)
+                             party=party, url=link,
+                             photo_url=photo_url)
 
             leg.add_office('capitol', 'Capitol Office', address=address,
                            phone=phone, email=email)
 
             if position:
-                leg.add_role( position, term, chamber=chamber,
+                leg.add_role(position, term, chamber=chamber,
                              district=district, party=party)
 
             leg.add_source(url)
 
-            #Probably just get this from the committee scraper
-            #self.scrape_member_page(link, session, chamber, leg)
+            # Probably just get this from the committee scraper
+            # self.scrape_member_page(link, session, chamber, leg)
             self.save_legislator(leg)
 
     def scrape_member_page(self, url, session, chamber, leg):
         html = self.get(url).text
         root = html.fromstring(html)
-        #get the committee membership
+        # get the committee membership
         c = root.xpath('//td/div/strong[contains(text(), "Committee")]')
         for row in c.xpath('ancestor::table[1]')[1:]:
             name = row[0].text_content().strip()


Test patch:
Repository: openstates/openstates-scrapers

Issue:
MO Committee Members are incorrect
https://openstates.org/api/v1//committees/MOC000028/?state=mo&apikey=xxxxx
vs
http://house.mo.gov/CommitteeList.aspx (Budget Committee)

Some don't even show in the Legislators list such as:
Sue Allen
Eric Burlison
Lincoln Hough
and Caleb Jones


Discussion:
scraper needs to be revisited, is failing quickly

Resolving patch:
diff --git a/openstates/mo/committees.py b/openstates/mo/committees.py
--- a/openstates/mo/committees.py
+++ b/openstates/mo/committees.py
@@ -21,6 +21,10 @@ def scrape(self, chamber, term):
         sessions = term.split('-')
 
         for session in sessions:
+            # The second session in a two year term might not have details yet
+            if session not in self.metadata['session_details']:
+                continue
+
             session_start_date = self.metadata['session_details'][session]\
                 ['start_date']
 
@@ -54,7 +58,8 @@ def _scrape_upper_chamber(self, session, chamber):
             '//div[@id = "{}"]//p/a'.format(comm_container_id))
 
         for comm_link in comm_links:
-            if "Assigned bills" in comm_link.text_content():
+            # Normalize to uppercase - varies between "Assigned bills" and "Assigned Bills"
+            if "ASSIGNED BILLS" in comm_link.text_content().upper():
                 continue
 
             comm_link = comm_link.attrib['href']


Test patch: