benoitc/gunicorn — pull request #342 by djoume (2012-05-09 17:01:15+00:00)
Title: gunicorn bad performance with file uploads and webob Request

Hi there,

We recently notice a performance issue with gunicorn when dealing with file uploads. The problem can be reproduced with this minimal wsgi app:

wsgi_app.py
8<--------------------------------------->8
from webob import Request

def app(environ, start_response):
if environ['REQUEST_METHOD'] == 'POST':
start_response('200 OK', [('content-type', 'text/html')])
req = Request(environ)
return ["DONE - ", '%s' % ('data' in req.POST)]
else:
start_response('200 OK', [('content-type', 'text/html')])
return ['

Test:

']

def app_factory(global_config, **local_conf):
return app
8<--------------------------------------->8

gunicorn.ini
8<--------------------------------------->8
[server:main]
use = egg:gunicorn#main
workers = 1
host = 0.0.0.0
port = 5000
timeout = 300

[app:main]
paste.app_factory = wsgi_app:app_factory
8<--------------------------------------->8

paste.ini
8<--------------------------------------->8
[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 5000

[app:main]
paste.app_factory = wsgi_app:app_factory
8<--------------------------------------->8

I have timed uploading a 10MB file using:

time curl -X POST 'http://localhost:5000' -F "data=@/tmp/10MB.data;type=application/octet-stream"

Using paste.ini it takes ~1 seconds, using gunicorn.ini is takes > 60 seconds

Also CPU usage is low when using paste and very high when using gunicorn.

The issue is due to gunicorn.http.body.Body.readline() reading one byte at a time using self.reader.read(1) line 243

I have fixed the issue by using the body buffer to minimize the number of calls to read().

Also the current implementation of readline() could return more than the number of bytes requested when the buffer is loaded (see tests), this is fixed as well.