benoitc/gunicorn — pull request #460 by hongqn (2012-12-25 17:20:12+00:00)
Title: optimize readline()

Use less memory when processing file uploading, to be fast and safe.

cgi.py in stdlib call fp's `readline()` with parameter `size=1<<16`. This cause huge memory usage when uploading large files in current implementation of `gunicorn.http.body.Body.readline()` and very slow (~140 seconds for 10MB file uploaded in my situation).

#342 optimized readline with buffered read. However, that's not enough. webob has it's own read buffer, but not every framework has that, e.g. web.py.

The following code shows the problem:

```python
# wsgi_app.py
import web

urls = ('/', 'upload')

class upload:
def POST(self):
x = web.input(data={})
data = x['data'].file.read()
return "%d bytes uploaded" % len(data)

app = web.application(urls, globals())
wsgi_app = app.wsgifunc()

if __name__ == '__main__':
app.run()
```

Run with gunicorn:

```sh
$ gunicorn wsgi_app:wsgi_app
```

Upload a file with 10MB size:
```sh
$ curl -X POST 'http://localhost:8000/' -F "data=@/Users/hongqn/tmp/10M;type=application/octet-stream"
```

In my laptop the uploading failed because the worker timed out.

Compare to use web.py's builtin server:

```sh
$ python wsgi_app.py 8000
```

It finished in ~500ms.

This pull request rewrotes readline() using a buffer with 1024 bytes. It take much less memory and finish the upload in ~500ms too.

I changed StringIO in six.py to cStringIO for better performance, btw.