"OverflowError: string longer than 2147483647 bytes" when trying requests.put

Hi,

I'm trying to upload a file that weight about 3GB and I'm getting the following error:
"OverflowError: string longer than 2147483647 bytes"

If I understand correctly it seems like there's a 2GB limit? didnt manage to find any reference to such limiation or how to bypass it (if possible). 

The code i'm using is:

``` python

datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
    args = myfile.read()
resp = requests.put(url, data=args, verify=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 99, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python2.7/httplib.py", line 995, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1029, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 991, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 844, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 820, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/ssl.py", line 234, in sendall
    v = self.send(data[count:])
  File "/usr/local/lib/python2.7/ssl.py", line 203, in send
    v = self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes
```

For smaller files this code works fine for me.


Rather than reading the entire file and sending it across in a single request, would it be possible for you to use chunked transfer encoding? http://docs.python-requests.org/en/latest/user/advanced/#chunk-encoded-requests


This limitation is in `httplib`. You can easily avoid it by slightly changing your code:

``` python
datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
    resp = requests.put(url, data=myfile, verify=False)
```


@Lukasa that's inaccurate. The traceback comes from an SSL wrapped socket. This has nothing to do with httplib from what I can see. `OverflowErrors` are raised when a value is larger than the underlying C integer size allowed. This can be seen if you call `len(something_larger_than_four_gb)` on a 32 bit system.


Unfortunately it looks like it cannot be avoided when you do a POST request with several headers. Then the file (or the files) is always read completely.

It would be great when this could be avoided in requests since I often have to send files which are longer than the available main memory on the system.


I'm just gonna chime in.

If you're trying to send files via the Web that are larger than your
systems memory can handle, HTTP probably isn't the best protocol to do this
via.

On Tue, 11 Oct 2016, 5:54 AM Erik Tews notifications@github.com wrote:

> Unfortunately it looks like it cannot be avoided when you do a POST
> request with several headers. Then the file (or the files) is always read
> completely.
> 
> It would be great when this could be avoided in requests since I often
> have to send files which are longer than the available main memory on the
> system.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/kennethreitz/requests/issues/2717#issuecomment-252729478,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AGk_7XIpx0UDRPXDuoopI7jDSfAlurgLks5qypf7gaJpZM4FqFGb
> .


That's true, but I don't get to decide on the protocol and endpoint.

Doing the request with curl works fine, and as a workaround, I'm currently printing a curl command to STDOUT so that the user can launch it.


@eriktews can you share how you're doing the upload? There are ways to stream uploads (like [Lukasa's comment](https://github.com/kennethreitz/requests/issues/2717#issuecomment-130343655) shows). Is there a reason you cannot do that (if you are not already trying that)? Also, can you provide your actual traceback?


So the comment from Lukasa seems to work when you are uploading a single file, then you can do a streaming upload. But I have to do a normal post request with several variables in the data part and the file as a part of a multipart upload.

There is an API documentation at https://canvas.instructure.com/doc/api/file.file_uploads.html which shows a curl command in the "Step 2" section. Basically I wanna replicate that call with the requests package in streaming mode.

I don't have the traceback at the moment, but when I get it, I will post it here.


Have you tried using the `MultipartEncoder` from the [requests-toolbelt](https://toolbelt.readthedocs.io)? That allows you to stream multipart/form-data uploads like doing an upload of a single file. It was written specifically for this use case and was added to our docs.


No, but that looks like right what I need. I will give it a try and see whether this works. I didn't know about the toolbelt package at all, so maybe you should reference it in the normal requests package documentation.


> so maybe you should reference it in the normal requests package documentation.

It is :)


@Lukasa 's method does not work - even with httplib off the signing still happens for the transport itself.  In my case I have a 2GB+ POST request (not a file, just POST data).  This is for an elasticsearch bulk update.  The endpoint only has HTTPS so I'm working through other solutions now.

```
Traceback (most recent call last):
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/elasticsearch/connection/http_requests.py", line 75, in perform_request
    timeout=timeout or self.timeout, verify=False)
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/virtualenvs/centr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 356, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 891, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 861, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 586, in write
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes
```

Sorry, can you demonstrate your code please?

This should throw the error:
```
datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
    r = requests.post(endpoint, data={'key': myfile.read()}, verify=False)
```

If `endpoint` is https then `ssl` will have to process the payload.  I wonder if `requests` or `requests-toolbelt` could have an option to do the signature with some other library that doesn't die when signing a 2GB string.  Of course, I would say that people shouldn't be signing such large things but it's definitely a real crash that's happening in the real world.

@adamn That was not my proposed solution. My proposed solution was to not read the file in manually at all. You are bumping into the same error as before, which is that we are sending a single gigantic string to httplib.

This is a behaviour we can fix: if we spot someone uploading a *gigantic* single string via Python then we can resolve it. But at this point I *strongly* recommend you use an intermediary file object: either one on disk, or by doing the urlencoding yourself and wrapping the result in a `BytesIO`.

I've already come up with a workaround so won't be able to dig deeper into this unfortunately.  I still suspect that the SSL payload needs to be signed/encrypted so the same thing will happen regardless of whether there is a file object or not since the exception is raised by `ssl.write` itself and I presume that method needs the entire payload.  Chunking the POST seems like the only real option.  Anyway, thanks for the help.

@adamn No, that's not necessary. TLS uses stream encryption, it does not need the entire payload at once.

What you're missing is that when given a file object, requests will automatically stream it in smaller chunks (specifically, 8192 kb chunks). Those cause no problem.

Sorry to comment on an old issue, but this looks _similar_ to an issue we've run into and I'm trying to decide whether it's worth opening a new issue for it.

Again, `requests.put` where data is a _huge_ string doesn't work, but we don't get an error. requests just hangs sending; a packet capture shows that no more data is being sent.

This behaviour is _worse_ that an exception being raised.

Has the remote peer been ACKing at the TCP level? Is it still reading from the receive buffer? Has it TCP FIN'd?

Yes, the remote end is sending ACKs appropriately, no FIN or anything like that.

In fact, if you have a large file `f`, we see the problem if you do `requests.put(url, data=f.read())` but not if you do `requests.put(url, data=f)`. Obviously if we have a file handle, we wouldn't bother to call read on it, but the point is that both calls are supposed generate the same request, and a packet capture shows that they do up until the point at which one stops sending packets.

Hrm. Is it possible for you to put together a small repro scenario? Do you see the same effect with other hosts?

As luck would have it, I have already done so.

Github doesn't seem to want to let me attach files, so:

```
#!/usr/bin/env python

import requests


MB = 1024 ** 2
GB = MB * 1024


if __name__ == '__main__':
    data = 'x' * 4 * GB
    resp = requests.put('http://localhost:8000', data=data)
    print resp
```

And a server to run it against:

```
#!/usr/bin/env python

import BaseHTTPServer
import logging


READ_CHUNK = 1024 ** 2


class Handler(BaseHTTPServer.BaseHTTPRequestHandler):
    def do_PUT(self):
        logging.info("PUT request recieved")
        for header, value in self.headers.items():
            logging.info("header: %s = %s", header, value)

        length = int(self.headers['Content-Length'])

        logging.info("Content length %s, getting content...", length)

        while length:
            to_read = min(length, READ_CHUNK)
            logging.info("reading %s bytes...", to_read)
            self.rfile.read(to_read)
            length -= to_read

        logging.info("Recieved content")

        self.send_response(200)


def run(server_class=BaseHTTPServer.HTTPServer):
    server_address = ('', 8000)
    httpd = server_class(server_address, Handler)
    httpd.serve_forever()


if __name__ == '__main__':
    logging.basicConfig(
        level=logging.DEBUG,
        format="%(asctime)s %(levelname)s: %(message)s",
    )
    logging.debug("Starting server")
    run()
```

Obviously this isn't the server we were running against when we first encountered this problem :)

Well as a first logical note I should point out that this is necessarily not the same problem as originally reported on this issue, as the original report affected TLS only, as discussed above. :wink: Regardless, let's dig into this a bit.

Ah, sorry. Is it worth me opening a new issue then, or should I just leave it here, since you're already looking at it?

Let's leave it here for now. =)

Huh. That behaves...very oddly. On my machine, over the loopback, I don't see any data sent at all: it's like Requests just gave up on sending it. Further debugging seems to show this is happening at the level of `socket.sendall`, which for some absurd reason is just not sending the complete response. By "not sending the complete response" I mean `socket.sendall` is *returning* early, but demonstrably is not sending all the data.

Naturally, the reason this happens is the same as the reason Python does lots of other stupid crap: `socket.sendall` is written in C. The *very first* thing that `socket.sendall` does is get the length of the data that was sent into it and shoves it into a C `int`. Now, this is wrong to begin with: `Py_buffer.len` is a `Py_ssize_t`, and `sizeof(ssize_t)` is *frequently* larger than `sizeof(int)`. So that's bonkers stupid, and probably the source of this bug.

In fact, it definitely is, since the current Python master has a changed `sendall` that uses the correct size. This seems to have been cleaned up around Python 3 time as a general "64-bit issue" (see python/cpython@19467d27ff14ebe31978438078ed5a661ffd29fb) in the socket module.

That makes this ultimately a duplicate of [CPython issue #18100](https://bugs.python.org/issue18100). This has been open a long time in need of patch review, and given that Python 2.7 is now only getting security fixes I doubt the CPython developers will fix it at this point.

This is a difficult issue for Requests to sensibly police. We can tell when people will *definitely* hit it (e.g. because the input is a string which a length greater than 2GB), but there are many situations where people will hit it but we can't tell (e.g. because the string plus the headers is greater than 2GB in size, or because there is a different type in use that CPython will treat as "stringish" that is larger than 2GB). So my initial inclination is, given that this is an issue that can be solved by moving to a newer version of Python, and that it can be worked around by not reading gigantic strings into memory (which is a best-practice anyway), and that if we ever move off httplib we'll fix it automatically anyway, I'm inclined to suggest that we probably don't have a huge pressure to resolve the issue? For my part, I think this is getting pretty close to "Dr, it hurts when I do this." "So don't do that then!" territory.

However, I'm willing to be disagreed with here.

The workaround is really simple - just wrap it in StringIO/BytesIO, but when you run into it, it's difficult to diagnose, so any help that requests could give, even if it's just a warning in the documentation, would be appreciated.

I can get behind the idea of a PR that adds a note in the documentation.

Running into the same problem,I dont this thread has an accepted solution,anyone has a solution for this ?

This also results in an OverflowError in `self._sslobj.write(data)`:

    files = {'file': open(tar_file_path, 'rb')}
    headers = {'key': 'abc123'}
    r = requests.post(url, files=files, headers=headers)

The file is 3GB in size.