Unable to upload chunks to Flask

I do not know if it comes from flask or requests , I link this issue posted on Flask issues

https://github.com/mitsuhiko/flask/issues/1668


Are you sure the headers you posted in the Flask issue are right? They look totally invalid. 


Yes, these headers are received by Flask.
I'm working on localhost, request between "requests" and "Flask" are send directly.

However I just tested this same case on a server with (flask / uwsgi / nginx and it works,uwsgi seems to put the content in a buffer before writing but that's another problem

Is this a content decoding problem between requests and flask ? I don't know yet.

For information on my server (flask / nginx / uwsgi) I have the same header (but Content-Length is set):

> Transfer-Encoding: chunked
> Content-Length: 30617496
> User-Agent: python-requests/2.8.1
> Connection: keep-alive
> Host: xxx.com
> Accept: _/_
> Content-Type: 
> Accept-Encoding: gzip, deflate


That data is also wrong: transfer-encoding chunked should not be sent with content-length, really.

I suspect Flask's built-in web server may have problems with chunked transfer encoding. Why are you using the generator instead of passing the file object directly to requests?


I planned to attach a progress indicator and it is not possible if I pass the file object.


@yoyoprs Sure it is.

``` python
class ProgressFileWrapper(object):
    def __init__(self, file):
        self._file = file

    def read(self, amt=None):
        print "reading"
        return self._file.read(amt)

    def __getattr__(self, name):
        return getattr(self._file, name)
```


if i do this as the advanced example:

``` python
with open(localpath, 'rb') as f:
    r = self.session.post(url, data=f, allow_redirects=False, params=params)
```

i lost the chunked transfer encoding:

> Content-Length: 25468
> User-Agent: python-requests/2.8.1
> Connection: keep-alive
> Host: xxx.com
> Accept: _/_
> Content-Type: 
> Accept-Encoding: gzip, deflate


@yoyoprs That's because that's the defined behaviour of requests. This is why I have suggested using file objects directly, or via the wrapping proxy defined above.


@Lukasa I do not understand the usefulness of the wrapper (except for progression).

My transfer is not streamed, (except if I control the sending of blocks in the proxy object, we agree) but the server does not receive header that tells it that the transfer is chunked. Should I force the header before send the request? Without this header, it will be complicated for components like uwsgi to understand the request and to avoid content buffering (although in my case for now).

It is not stated anywhere in the documentation it is not possible to stream with post method the contents of a file (using a generator or not) and I still do not understand why Flask (without nginx and uwsgi) can not read the stream


@yoyoprs I don't understand what you're talking about.

A file object passed to the `data` argument _is_ streamed, as the documentation [clearly states](http://docs.python-requests.org/en/latest/user/advanced/#streaming-uploads). The exact chunk of code you posted above streams the file.

In this context, what we mean by _streamed_ is that Requests does not load the file entirely into memory: instead, we loop over the file sending 8kB worth of data with each send to the socket.

What you're talking about is not streaming, it is _chunked encoding_. Chunked encoding is a different notion: it refers to the way the data is constructed and sent on the wire. Once again, it is possible to do this: sending a generator does that exactly. This is why I believe your problem is not with requests: I believe that someone in your _server_ stack is adding in a content-length header. Requests most certainly is not. To prove it, let me use the following script to upload some data to http2bin:

``` python
import requests

def gen():
    with open('/path/to/large/logfile', 'rb') as f:
        while True:
            data = f.read(1024)
            if not data:
                break
            yield data
session = requests.Session()
r = session.post('http://http2bin.org/post', data=gen())
print r.status_code
print r.content
```

I captured the data I sent using Wireshark, which records _exactly_ the bytes on the wire, without any application processing. This says my request looked like this:

```
POST /post HTTP/1.1
Host: http2bin.org
Transfer-Encoding: chunked
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.9.0

400
machine-0: 2015-06-28 14:06:00 INFO juju.cmd supercommand.go:37 running jujud [1.24.0-trusty-amd64 gc]
machine-0: 2015-06-28 14:06:00 DEBUG juju.agent agent.go:425 read agent config, format "1.18"
machine-0: 2015-06-28 14:06:00 INFO juju.cmd.jujud machine.go:419 machine agent machine-0 start (1.24.0-trusty-amd64 [gc])
machine-0: 2015-06-28 14:06:00 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/juju/wrench: no such file or directory
machine-0: 2015-06-28 14:06:00 INFO juju.cmd.jujud upgrade.go:87 no upgrade steps required or upgrade steps for 1.24.0 have already been run.
machine-0: 2015-06-28 14:06:00 INFO juju.network network.go:194 setting prefer-ipv6 to false
machine-0: 2015-06-28 14:06:00 INFO juju.worker runner.go:261 start "api"
machine-0: 2015-06-28 14:06:00 INFO juju.worker runner.go:261 start "statestarter"
machine-0: 2015-06-28 14:06:00 INFO juju.api apiclient.go:331 dialing "wss://localhost:17070/environment/adad29a3-1105-4483-863b-9496d5495c82/api"
machine-
400
0: 2015-06-28 14:06:00 INFO juju.worker runner.go:261 start "termination"
machine-0: 2015-06-28 14:06:00 INFO juju.worker runner.go:261 start "state"
machine-0: 2015-06-28 14:06:00 INFO juju.api apiclient.go:339 error dialing "wss://localhost:17070/environment/adad29a3-1105-4483-863b-9496d5495c82/api": websocket.Dial wss://localhost:17070/environment/adad29a3-1105-4483-863b-9496d5495c82/api: dial tcp 127.0.0.1:17070: connection refused
machine-0: 2015-06-28 14:06:00 ERROR juju.worker runner.go:219 exited "api": unable to connect to "wss://localhost:17070/environment/adad29a3-1105-4483-863b-9496d5495c82/api"
machine-0: 2015-06-28 14:06:00 INFO juju.worker runner.go:253 restarting "api" in 3s
machine-0: 2015-06-28 14:06:00 DEBUG juju.service discovery.go:118 discovered init system "upstart" from local host
machine-0: 2015-06-28 14:06:00 DEBUG juju.cmd.jujud machine.go:1312 mongodb service is installed
machine-0: 2015-06-28 14:06:00 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.

[ many more chunks of data stripped]
```

This is a chunked transfer encoded body, as shown by the headers. There is no content length. The body is correctly encoded.

Requests is _not_ getting this wrong. This is why I asked about your headers.

Note, however, that WSGI provides no specific guidance on how a WSGI server should handle chunked uploads. Therefore, I recommend you look at the interface between uWSGI and Flask.


@Lukasa sorry for my broken English, on the flask side issue I mention "chunked" and not on this one, my mistake.
What interests me is to successfully write directly content of chunked transfer on Flask side and so now I know it is not "requests" the problem (i was redirected here ...)

There is a good chance that Flask adds this header, but it does not explain why without uwsgi interface (with it works), it is impossible to read the stream (Flask would require the Content-Length to start streaming ? Maybe ...)

I will make several tests (without/with uwsgi interface) and try this: http://uwsgi-docs.readthedocs.org/en/latest/Transformations.html#streaming-vs-buffering

Thank you for your patience really.


@yoyoprs do I understand correctly that we can close this now?


@sigmavirus24 yes you can :)


Okay @yoyoprs. I'll be in #python-requests on chat.freenode.net (IRC network) if you want to some one to chat with about this while you're debugging.


@Lukasa Just a question off issue ... with this method:

``` python
with open(localpath, 'rb') as f:
    r = self.session.post(url, data=f, allow_redirects=False, params=params)
```

Do you have a bug when the file size is 0 byte ?


We did: [it was fixed in 2.9.0](https://github.com/kennethreitz/requests/blob/master/HISTORY.rst#290-2015-12-15).


I use 2.9.1 ...


Did you update? Because the headers you posted above said you were using 2.8.1. If you did update to 2.9.1, what problem are you seeing?


Yes i have updated. I get a ConnectionError when file size is 0 byte.


What ConnectionError?


 File "C:\Python27\lib\site-packages\requests\sessions.py", line 511, in post
    return self.request('POST', url, data=data, json=json, *_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
    resp = self.send(prep, *_send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 426, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))


So this suggests the server doesn't like the request at all. Do you have any logs from your server?


On server side (Timeout):  "POST /api/.../ HTTP/1.1" 408 0 "-" "python-requests/2.9.1"
With others files (where size is > 0), it works fine. 


408 is a weird request timeout issue, so it seems like something in your server stack is getting this wrong. Do you want to confirm that your flask app will handle this properly?

Requests is sending a totally valid body in this case, specifically an empty chunked encoded body.

```
POST /post HTTP/1.1
Host: http2bin.org
Transfer-Encoding: chunked
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.9.0

0

```

This means that the problem is server-side.


I just tested, everything works with version 2.8.1

I think the answer is:
http://nginx.org/en/docs/http/ngx_http_uwsgi_module.html#uwsgi_request_buffering
uwsgi_request_buffering is off on my nginx, to allow Flask write directly without pre-buffer data ...

When you turn on Transfer-Encoding Chunked in 2.9.1, by default nginx want to put content in buffer ... and there is nothing to put in buffer (because no content) so he waits until the timeout ...


nginx should be able to cope with an empty chunked body, and if it can't that's a bug with nginx.


That seems normal that nginx wait chunked data, it does not know the content length.
There is a way to delete header "Transfer-Encoding" before send the post to avoid this type of case ?


@yoyoprs This is _not_ normal. The way to signal the end of a chunked body is with a zero-length chunk (specifically, the bytes `0\r\n\r\n`). Requests has done that: the body is signaled to be complete, and it just happens to have zero length.

nginx _must_ cope with this, it's a specification violation for it not to.


Thank's for these details.
Would there be a workaround ?


Yes: do not send zero-length files. =) If `requests.utils.super_len()` returns `0` when passed the file object, simply don't send it.


Or stay with 2.8.1 for life: p
I reach my goals with a prepared query, thank you gain ;)