How to upload binary file using PUT?

Hi,

I just started using requests recently - it's a big improvement over the urllib2 library!  But I'm having some difficulty with uploading files and I hope somebody can help me figure it out.

I'm trying to use a REST API where I need to upload a binary file to the server.  The API uses https and Kerberos authentication (I'm also using the requests_kerberos package).

With curl, I can do this:

```
curl --negotiate -u : -H "X-Codesigner-security-token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --upload-file <pathname> https://myservice.acme.com/api/v1/requests/24449/unsigned
```

Which results in this:

```
=> Send header, 282 bytes (0x11a)
0000: PUT /api/v1/requests/24449/unsigned HTTP/1.1
002e: User-Agent: curl/7.21.6 (i386-pc-win32) libcurl/7.21.6 OpenSSL/0
006e: .9.8r zlib/1.2.5
0080: Host: myservice.acme.com
009d: Accept: */*
00aa: X-Codesigner-security-token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
00e9: Content-Length: 4560799
0102: Expect: 100-continue
0118: 
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 16384 bytes (0x4000)
…
=> Send data, 16384 bytes (0x4000)
…
```

With requests, I've tried using both streaming upload and chunked upload, but haven't been able to make them work.  I'm using the latest master (commit c63472e328946f464ed9fa24c0fadd2890966c5b). 

Code snippet using chunked transfer:

```
def filegen(fo):
    yield fo.read(CHUNK_SIZE)

def main():
    session = requests.Session()
    session.auth = HTTPKerberosAuth()

    upload_headers = {'X-Codesigner-security-token': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'}

    with open(pathname, 'rb') as infile:
        response = session.put(upload_url, headers=upload_headers, data=filegen(infile))
```

This results in:

```
Request headers: {
'Accept-Encoding': 'gzip, deflate, compress', 
'X-Codesigner-security-token': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 
'Accept': '*/*', 
'User-Agent': 'python-requests/1.1.0 CPython/2.7.1 Windows/7', 
'Transfer-Encoding': 'chunked', 
'Authorization': 'Negotiate <blah>'
}

Response headers: {
'status': '500', 
'content-length': '441', 
'content-encoding': 'gzip', 
'x-powered-by': 'Phusion Passenger (mod_rails/mod_rack) 3.0.11', 
'vary': 'Accept-Encoding,User-Agent', 
'connection': 'close', 
'server': 'Server', 
'x-runtime': '0.564621', 
'date': 'Tue, 26 Mar 2013 18:29:44 GMT',
'content-type': 'text/html', 
'www-authenticate': 'Negotiate <blah>'
}
```

Using the streaming upload:

```
with open(pathname, 'rb') as infile:
    response = session.put(upload_url, headers=upload_headers, data=infile)
```

Results in:

```
Request headers:{
'X-Codesigner-security-token': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 
'Content-Length': u'20319', 
'Accept-Encoding': 'gzip, deflate, compress', 
'Accept': '*/*', 
'User-Agent': 'python-requests/1.1.0 CPython/2.7.1 Windows/7'
}

2013-03-26 12:16:04,440 DEBUG "PUT /api/v1/requests/24539/unsigned HTTP/1.1" 401 401

Traceback (most recent call last):
  ...
  File "C:\Python27\lib\site-packages\requests-1.1.0-py2.7.egg\requests\sessions.py", line 376, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "C:\Python27\lib\site-packages\requests-1.1.0-py2.7.egg\requests\sessions.py", line 321, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests-1.1.0-py2.7.egg\requests\sessions.py", line 426, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "C:\Python27\lib\site-packages\requests-1.1.0-py2.7.egg\requests\hooks.py", line 41, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "build\bdist.win32\egg\requests_kerberos\kerberos_.py", line 236, in handle_response
  File "build\bdist.win32\egg\requests_kerberos\kerberos_.py", line 158, in handle_401
  File "build\bdist.win32\egg\requests_kerberos\kerberos_.py", line 147, in authenticate_user
  File "C:\Python27\lib\site-packages\requests-1.1.0-py2.7.egg\requests\adapters.py", line 246, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError:
 HTTPSConnectionPool(host='myservice.acme.com', port=443): Max retries exceeded with url: /api/v1/requests/24539/unsigned (Caused by <class 'socket.error'>: [Errno 10054] An existing connection was forcibly closed by the remote host)
```


This might actually be an issue with `requests_kerberos`. I completely forgot to push our latest version, which I'm about to do now. If you could upgrade after that and see if this persists, I would appreciate it.


Cool, thanks!  I'll pull the latest requests_kerberos after it's been pushed.

I might also have a small contribution for requests_kerberos to make it work with the Windows [kerberos-sspi](https://pypi.python.org/pypi/kerberos-sspi/0.1) package as well as the standard kerberos package.


Version 0.2 is on PyPI so feel free to upgrade. And we'd love pull requests over at the kerberos repo.


Thanks for the quick response!  Unfortunately, requests_kerberos 0.2 didn't fix the issue for me (I had actually pulled the requests_kerberos source from github and built it locally before you posted 0.2 to PyPI, so it looks like I was essentially using 0.2 already).

I've done a little investigation in the requests sources, specifically in the models.py file.  A few questions / observations:

In models.py line 351:

```
    try:
        length = str(super_len(data))
```

If `super_len(data)` returns `None`, then `length == u'None'`, not `None`, which is probably not what was intended.

That comes into play around line 362:

```
        if length:
            self.headers['Content-Length'] = length
        else:
            self.headers['Transfer-Encoding'] = 'chunked'
```

Since `length == u'None'`, the `Content-Length` header gets set to the string 'None', instead of the `Transfer-Encoding` header being set to 'chunked.'

Another question involves the `prepare_auth()` function. at lines 418-419:

```
        # Recompute Content-Length
        self.prepare_content_length(self.body)
```

Since it calls `self.prepare_content_length()` unconditionally, it could result in a different Content-Length than was set in `prepare_body()`.

I did build a version of requests that attempts to address the behavior above.  I managed to get the PUT request to use 'Transfer-Encoding: chunked' instead of setting the Content-Length header (the output in the original post now shows the results), but it still raises the same exception (just a slightly different stack trace).

I can do a pull request with my changes if that helps diagnose the problem.


So I can't address the first two concerns you brought up from my phone, but the last one is not valid. There are cases where during authentication `self.body` is changed and in those cases the Content-Length header has to be regenerated. That must stay.

Can you give an example of when `super_len` returns `None` for when I can look at this later?


Hi,

The case where `super_len` returns `None` is when `data` is a generator. From the documentation, it looks like this is the way to make requests use [chunked encoding](http://docs.python-requests.org/en/latest/user/advanced/#chunk-encoded-requests).

Example:

```
def filegen(fo):
    yield fo.read(CHUNK_SIZE)

def main():
    ...
    with open(pathname, 'rb') as infile:
        response = session.put(upload_url, headers=upload_headers, data=filegen(infile))
```

Re: `prepare_auth` re-computing the Content-Length - I realize that's by design. It seems that it might need to check whether there's already a 'Transfer-Encoding: chunked' header, and if so, skip re-computing the content length (?). 

But I'm not an HTTP expert, so I don't know if those two headers are mutually exclusive.


Changes to fix this, according to your estimation of the problem @dabono, are in 1.2.0 which is out on PyPI and Crate. I'm closing this but if you still have this issue, let me know and I'll reopen the issue.