Repository: urllib3/urllib3

Issue:
OverflowError: signed integer is greater than maximum
Both of the below calls result in an `OverflowError` being raised because the `Content-Length` is longer than 3GB. OpenSSL only supports reading (2**31)-1 bytes in a single read call.

```python
url = "https://dailymed-data.nlm.nih.gov/public-release-files/dm_spl_release_human_rx_part1.zip"

resp = http.request("GET", url)
resp.data

resp = http.request("GET", url, preload_content=False)
list(resp.stream(None))
```

```
Traceback (most recent call last):
  File "/home/sethmlarson/urllib3/testme.py", line 5, in <module>
    resp = http.request("GET", "https://dailymed-data.nlm.nih.gov/public-release-files/dm_spl_release_human_rx_part1.zip")
  File "/home/sethmlarson/urllib3/src/urllib3/__init__.py", line 112, in request
    return _DEFAULT_POOL.request(
  File "/home/sethmlarson/urllib3/src/urllib3/_request_methods.py", line 109, in request
    return self.request_encode_url(
  File "/home/sethmlarson/urllib3/src/urllib3/_request_methods.py", line 142, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/home/sethmlarson/urllib3/src/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/sethmlarson/urllib3/src/urllib3/connectionpool.py", line 744, in urlopen
    response = self.ResponseCls.from_httplib(
  File "/home/sethmlarson/urllib3/src/urllib3/response.py", line 748, in from_httplib
    resp = ResponseCls(
  File "/home/sethmlarson/urllib3/src/urllib3/response.py", line 467, in __init__
    self._body = self.read(decode_content=decode_content)
  File "/home/sethmlarson/urllib3/src/urllib3/response.py", line 664, in read
    data = self._fp.read() if not fp_closed else b""
  File "/home/sethmlarson/.pyenv/versions/3.8.6/lib/python3.8/http/client.py", line 471, in read
    s = self._safe_read(self.length)
  File "/home/sethmlarson/.pyenv/versions/3.8.6/lib/python3.8/http/client.py", line 612, in _safe_read
    data = self.fp.read(amt)
  File "/home/sethmlarson/.pyenv/versions/3.8.6/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/home/sethmlarson/.pyenv/versions/3.8.6/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/home/sethmlarson/.pyenv/versions/3.8.6/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
OverflowError: signed integer is greater than maximum
```

The fix would likely need to go into `urllib3.response.HTTPResponse` to limit the amount of data requested in a single call to `http.client.HTTPResponse.read(amt)`.

Originally reported in https://github.com/psf/requests/issues/5969

### Minimum requirements

:dollar: **You can get paid to complete this issue! [Please read the docs for more information](https://urllib3.readthedocs.io/en/latest/contributing.html#getting-paid-for-your-contributions).**

- [x] Ensure that any call to `urllib3.response.HTTPResponse.read(...)` never results in a call to `.read(n)` with a value greater than `n=(2**31)-1`. Behavior should not change for users, you should buffer bytes into a `bytearray` and then return the stitched together data when necessary.
- [x] Test cases for streaming chunk sizes, `.read(2**31)`, `.read(None)`, `.read()`, `preload_content=True`
- [x] Merge fix into both `main` and `1.26.x` branches


Discussion:
Thanks for bringing this up! Excited to see a solution, we've been hitting this as well. In the meanwhile does anyone happen to have any reasonable workarounds?
The best solution is to stream the response data instead of loading all at once. You'll likely want to do this anyways (even with the fix to this issue) as loading 3GB into memory all at once isn't likely to be performant.
I've looked a bit into the issue and IMHO changing the logic of `HTTPResponse.read()` would be wrong. 

First, it's currently possible to call `.read()` with `amt` larger than `2**31` - if you're using HTTP instead of HTTPS. For instance, if you replace the URL in the OP with `url = "http://speed.hetzner.de/10GB.bin"`, things seem to work as intended. And `HTTPResponse.read()` has no way of guessing what the underlying connection supports.

I guess that the issue here is mainly due to the obscure error message. From my (limited) understanding of the respective scopes of urllib3 and requests, I'd say that `urllib3.request('GET', HUGE_FILE)` is user error, unlike `requests.get(HUGE_FILE)` which should just work. Additionally, the behaviour of `.stream(None)` is another source of confusion, since it just wraps the result of `.read()` in a one-element iterator instead of acting like `.stream()` as I would have expected.
When CPython is used, the error should only happen since 3.8.0a4 (https://github.com/python/cpython/issues/80231, https://github.com/python/cpython/commit/d6bf6f2d0c83f0c64ce86e7b9340278627798090), it was fixed in 3.9.7 (https://github.com/python/cpython/issues/87019, https://github.com/python/cpython/commit/153365d864c411f6fb523efa752ccb3497d815ca), the fix was not backported to 3.8.

In the course of https://github.com/python/cpython/issues/87020, [one more regression](https://twitter.com/geofft/status/1417167982665551877) was introduced in 3.10.0b1 (https://github.com/python/cpython/commit/89d1550d14ba689af12eeb726e4ff8ce73cee7e1) and fixed in 3.10.0rc1 (https://github.com/python/cpython/commit/5ec275758dbc307e9838e7038bfc3b5390950ea7).

Resolving patch:
diff --git a/src/urllib3/response.py b/src/urllib3/response.py
--- a/src/urllib3/response.py
+++ b/src/urllib3/response.py
@@ -2,6 +2,7 @@
 import json as _json
 import logging
 import re
+import sys
 import zlib
 from contextlib import contextmanager
 from http.client import HTTPMessage as _HttplibHTTPMessage
@@ -44,6 +45,7 @@
 except (AttributeError, ImportError, ValueError):  # Defensive:
     zstd = None
 
+from . import util
 from ._collections import HTTPHeaderDict
 from .connection import _TYPE_BODY, BaseSSLError, HTTPConnection, HTTPException
 from .exceptions import (
@@ -665,6 +667,54 @@ def _error_catcher(self) -> Generator[None, None, None]:
             if self._original_response and self._original_response.isclosed():
                 self.release_conn()
 
+    def _fp_read(self, amt: Optional[int] = None) -> bytes:
+        """
+        Read a response with the thought that reading the number of bytes
+        larger than can fit in a 32-bit int at a time via SSL in some
+        known cases leads to an overflow error that has to be prevented
+        if `amt` or `self.length_remaining` indicate that a problem may
+        happen.
+
+        The known cases:
+          * 3.8 <= CPython < 3.9.7 because of a bug
+            https://github.com/urllib3/urllib3/issues/2513#issuecomment-1152559900.
+          * urllib3 injected with pyOpenSSL-backed SSL-support.
+          * CPython < 3.10 only when `amt` does not fit 32-bit int.
+        """
+        assert self._fp
+        c_int_max = 2**31 - 1
+        if (
+            (
+                (amt and amt > c_int_max)
+                or (self.length_remaining and self.length_remaining > c_int_max)
+            )
+            and not util.IS_SECURETRANSPORT
+            and (util.IS_PYOPENSSL or sys.version_info < (3, 10))
+        ):
+            buffer = io.BytesIO()
+            # Besides `max_chunk_amt` being a maximum chunk size, it
+            # affects memory overhead of reading a response by this
+            # method in CPython.
+            # `c_int_max` equal to 2 GiB - 1 byte is the actual maximum
+            # chunk size that does not lead to an overflow error, but
+            # 256 MiB is a compromise.
+            max_chunk_amt = 2**28
+            while amt is None or amt != 0:
+                if amt is not None:
+                    chunk_amt = min(amt, max_chunk_amt)
+                    amt -= chunk_amt
+                else:
+                    chunk_amt = max_chunk_amt
+                data = self._fp.read(chunk_amt)
+                if not data:
+                    break
+                buffer.write(data)
+                del data  # to reduce peak memory usage by `max_chunk_amt`.
+            return buffer.getvalue()
+        else:
+            # StringIO doesn't like amt=None
+            return self._fp.read(amt) if amt is not None else self._fp.read()
+
     def read(
         self,
         amt: Optional[int] = None,
@@ -702,13 +752,11 @@ def read(
         fp_closed = getattr(self._fp, "closed", False)
 
         with self._error_catcher():
+            data = self._fp_read(amt) if not fp_closed else b""
             if amt is None:
-                # cStringIO doesn't like amt=None
-                data = self._fp.read() if not fp_closed else b""
                 flush_decoder = True
             else:
                 cache_content = False
-                data = self._fp.read(amt) if not fp_closed else b""
                 if (
                     amt != 0 and not data
                 ):  # Platform-specific: Buggy versions of Python.


Test patch:
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -8,6 +8,7 @@
 import shutil
 import socket
 import ssl
+import sys
 import tempfile
 import time
 from collections import OrderedDict
@@ -1491,6 +1492,63 @@ def socket_handler(listener: socket.socket) -> None:
                 pool.request("GET", "/", retries=False, timeout=LONG_TIMEOUT)
         assert server_closed.wait(LONG_TIMEOUT), "The socket was not terminated"
 
+    # SecureTransport can read only small pieces of data at the moment.
+    # https://github.com/urllib3/urllib3/pull/2674
+    @notSecureTransport()
+    @pytest.mark.skipif(
+        os.environ.get("CI") == "true" and sys.implementation.name == "pypy",
+        reason="too slow to run in CI",
+    )
+    @pytest.mark.parametrize(
+        "preload_content,read_amt", [(True, None), (False, None), (False, 2**31)]
+    )
+    def test_requesting_large_resources_via_ssl(
+        self, preload_content: bool, read_amt: Optional[int]
+    ) -> None:
+        """
+        Ensure that it is possible to read 2 GiB or more via an SSL
+        socket.
+        https://github.com/urllib3/urllib3/issues/2513
+        """
+        content_length = 2**31  # (`int` max value in C) + 1.
+        ssl_ready = Event()
+
+        def socket_handler(listener: socket.socket) -> None:
+            sock = listener.accept()[0]
+            ssl_sock = ssl.wrap_socket(
+                sock,
+                server_side=True,
+                keyfile=DEFAULT_CERTS["keyfile"],
+                certfile=DEFAULT_CERTS["certfile"],
+                ca_certs=DEFAULT_CA,
+            )
+            ssl_ready.set()
+
+            while not ssl_sock.recv(65536).endswith(b"\r\n\r\n"):
+                continue
+
+            ssl_sock.send(
+                b"HTTP/1.1 200 OK\r\n"
+                b"Content-Type: text/plain\r\n"
+                b"Content-Length: %d\r\n\r\n" % content_length
+            )
+
+            chunks = 2
+            for i in range(chunks):
+                ssl_sock.sendall(bytes(content_length // chunks))
+
+            ssl_sock.close()
+            sock.close()
+
+        self._start_server(socket_handler)
+        ssl_ready.wait(5)
+        with HTTPSConnectionPool(
+            self.host, self.port, ca_certs=DEFAULT_CA, retries=False
+        ) as pool:
+            response = pool.request("GET", "/", preload_content=preload_content)
+            data = response.data if preload_content else response.read(read_amt)
+            assert len(data) == content_length
+
 
 class TestErrorWrapping(SocketDummyServerTestCase):
     def test_bad_statusline(self) -> None: