Can't upload file larger than 2GB to webdav by requests

Dear Sir,
I can't upload file larger than 2GB to webdav server by requests, and the following is the error message. Any idea about how to solve this problem ? Thanks a lot for your help.

OS: SL5,SL6
Python version:2.6.6
requests version:2.1.0

raceback (most recent call last):
  File "upload.py", line 5, in <module>
  File "/opt/rucio/.venv/lib64/python2.6/site-packages/requests/api.py", line 99, in put
    return request('put', url, data=data, *_kwargs)
  File "/opt/rucio/.venv/lib64/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/opt/rucio/.venv/lib64/python2.6/site-packages/requests/sessions.py", line 361, in request
    resp = self.send(prep, *_send_kwargs)
  File "/opt/rucio/.venv/lib64/python2.6/site-packages/requests/sessions.py", line 464, in send
    r = adapter.send(request, *_kwargs)
  File "/opt/rucio/.venv/lib64/python2.6/site-packages/requests/adapters.py", line 363, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 5] _ssl.c:1217: Some I/O error occurred

Best Regards,
Evan Chang


Hi: can I see the line of code you're using to send the file?


Hi,
It's my simple code:
import requests
file=open('/home/evan/file_2.5G_01','rb')
data=file.read()
requests.put('https://my/webdav/server/file_2.5G_01', data=data, verify=False, cert=('/tmp/evan.proxy','/tmp/evan.proxy'))

Thanks,
Evan Chang


Can you try streaming the upload? Change your code to:

``` python
import requests
file=open('/home/evan/file_2.5G_01','rb')
requests.put('https://my/webdav/server/file_2.5G_01', data=file, verify=False, cert=('/tmp/evan.proxy','/tmp/evan.proxy'))
```


Hi,
Now I can upload my file to webdav server by the code you changed, but I saw the file size is 0
when I use "du -sh" to check it.

du -sh wj_rucio_2.5G_20.10483315.0
0       wj_rucio_2.5G_20.10483315.0

Best Regards,
Evan Chang


@wchang we do not change the size of the file.


Hi,
I mean I checked the file size after I upload my file to webdav server, and I found the size is 0,
which means there is something wrong during the file upload. I'm not sure it's requests problem 
or webdav's problem, any comment ? Thanks a lot for your help. 


Oh the size on the server side is 0? That could e from an old upload that failed. Could you delete it and try again?


Yes, the size on the server side is 0 and I tried more than three times. I can upload files larger than 2GB by curl successfully and I can also upload files < 2GB without problem by requests.put, but I can't upload files > 2GB successfully by requests.put. 

Thanks. 


By the way, this is the curl command I'm using, maybe it's helpful for debugging 

curl -E /my/certificate --capath /path/for/verifying -L https://hostname/of/my/webdav/server:443/path/file_2.5G_30 -T file_2.5G_30

Thanks


So I reduced that to the simplest form, which is

```
curl --upload-file=file -L https://httpbin.org/put
```

For those unfamiliar (as I was), `-T/--upload-file` sets the verb to put and streams the file. cURL also **expects** a 100 Continue response. (#713) It receives that and redirects to what you would expect, which is a 200 OK. (To test this out yourself just add the `-v` option to the command above to see the debugging information.

Unfortunately, however, I cannot find very good documentation as to how cURL handles the data. There's no `Content-Type` header set by the request, simply `Content-Length`, which requests also sets for you with the streaming upload.

Looking more closely over the issue, however, your first code snippet includes this:

```
cert=('/tmp/evan.proxy','/tmp/evan.proxy')
```

Are you behind a proxy while trying to upload the file? If so, is it possible that is causing the issues you're seeing? It might not seem likely but the fact of the matter is that cURL handles proxies far better than we do.

Can you try putting the file to `http://httpbin.org/put` and let us know what happens. Specifically check what JSON is returned.


Hi, I got this message when I tried to upload a 2.5GB file. I didn't get such message when I uploaded 1MB file.

/put
Traceback (most recent call last):
  File "upload.py", line 6, in <module>
    requests.put('https://httpbin.org/put', data=file)
  File "/asgc_ui_home/wjc00/python26/lib/python2.6/site-packages/requests/api.py", line 99, in put
    return request('put', url, data=data, *_kwargs)
  File "/asgc_ui_home/wjc00/python26/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/asgc_ui_home/wjc00/python26/lib/python2.6/site-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, *_send_kwargs)
  File "/asgc_ui_home/wjc00/python26/lib/python2.6/site-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, *_kwargs)
  File "/asgc_ui_home/wjc00/python26/lib/python2.6/site-packages/requests/adapters.py", line 372, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /put (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

Cheers,
Evan Chang


And here is a brief code
import requests
file=open('/path/evan_2.5G_01','rb')
requests.put('https://httpbin.org/put', data=file)


Seeing `Connection reset by peer` is setting off alarm bells for me, especially as it isn't the same exception as you hit originally.

2,5GB is an interesting number, only because it's very slightly larger than the maximum value of a 32-bit signed integer, and also not very far from the maximum memory allowed to a 32-bit process. Does curl have trouble when uploading the same file to httpbin?


Hi,
I got the message when I upload file to httpbin by curl:
$ curl --upload-file evan_2.5G_02 -L https://httpbin.org/put
<!DOCTYPE html>
    <html>
    <head>
      <style type="text/css">
        html, body, iframe { margin: 0; padding: 0; height: 100%; }
        iframe { display: block; width: 100%; border: none; }
      </style>
    <title>Application Error</title>
    </head>
    <body>
      <iframe src="//s3.amazonaws.com/heroku_pages/error.html">
        <p>Application Error</p>
      </iframe>
    </body>

And yes, 2.5GB is really an interesting number. Actually, I tried to upload a 2.1GB file to our webdav server by my original code (data=file.read()) but it failed . And I can upload a 2.0GB file without any problem by my original code. Our cpu is Intel Xeon L5520. 
Thanks.

Evan Chang


Sorry, it's the message I got from uploading file to httpbin by curl

```
<!DOCTYPE html>
    <html>
    <head>
      <style type="text/css">
        html, body, iframe { margin: 0; padding: 0; height: 100%; }
        iframe { display: block; width: 100%; border: none; }
      </style>
    <title>Application Error</title>
    </head>
    <body>
      <iframe src="//s3.amazonaws.com/heroku_pages/error.html">
        <p>Application Error</p>
      </iframe>
    </body>
```


@wchang you ignored my question about whether you're behind a proxy or not. That would not cause the Application Error you were seeing with httpbin, but it might explain the bizarre difference in behaviour between putting a 2.0 GB file and a >2.0 GB file.


Oh~ Sorry. Yes, I would like to bind a proxy. But how does it make difference in behavior between 2GB and >2GB?
I suppose the proxy is just for user's validation.


So I think we can't use httpbin as a target here: it's clearly running out of memory.


Why it's been closed ? The problem is still there ?


So now I sum up current questions
1. I tried to use streaming  but the file size shows 0 on webdav server, how can I solve this issue? 
2. The proxy we use is just for user's certificate, does it affect the file transfer as the file size larger than 2GB?
3. I tried to search the issue on stackoverflow and there are some comments, do you think it's helpful (I tried the  
   code but didn't success)
   http://stackoverflow.com/questions/13909900/progress-of-python-requests-post

Thanks again for the help.
Cheers
Evan Chang


> 1. The proxy we use is just for user's certificate, does it affect the file transfer as the file size larger than 2GB?

This is a _possibility_ but not a certainty

> 1. I tried to search the issue on stackoverflow and there are some comments, do you think it's helpful

That's a very old question and answer. I doubt it would be helpful.

> 1. I tried to use streaming but the file size shows 0 on webdav server, how can I solve this issue? 

This is what's most baffling to me. And I wonder if this may be influenced by the proxy.


Hi
I tried to upload my file to another webdav server which doesn't need certificate (proxy) and the file size on server side is still 0. Maybe it's not the proxy's problem.

It's a simple code:

```
with open('/my/path/file_2.5G_01') as file:
requests.put('http://hostname/path/file_2.5G_01', data=file, verify=False)
```

Thanks a lot.


Yeah, I'm planning to try this locally and see what happens.


Yup, so this isn't a Requests bug. I set up this simple Flask app:

``` python
from flask import Flask, request
app = Flask(__name__)

@app.route("/", methods=["POST"])
def hello():
    with open('posted.dat', 'wb') as f:
        data = request.stream.read(1024)

        while data:
            f.write(data)
            data = request.stream.read(1024)

        return "Success!"


if __name__ == '__main__':
    app.run()
```

And then posted a 4.34GB Fedora ISO to it. No bug was encountered here, everything went off fine, and the MD5 hashes of the two files match.

This makes me think the proxy is to blame here.


If not the proxy then the server. But I found it hard to believe this could be a requests issue when it was returning 200 OK and the server didn't have the proper file size.