GoodCloud/django-ajax-uploader — Issue #12: Memory usage with bigger files
Author: deschler
Created: 2011-11-17 12:53:05+00:00
State: open

Uploading huge files seems to be problematic. I've tried to upload a 3GB file and the apache wsgi process slowly took up the entire 12GB RAM of the server. Using the cancel link didn't stop it and i had to restart apache in order to prevent further swapping.

Now i understand that http was never designed to handle huge uploads, but i'm wondering why the process eats up that much of memory. While the file is being uploaded, the RAM usage slowly increases by the amount of the filesize, which is fine i think. After it reaches 100% upload process however, the memory usage goes up further and the file is not written.

Using the django dev server, this behaviour is even more visible. Here a file of about 200MB is enough to make my machine with 4GB RAM swap.

I override upload_complete in my custom Backend which moves the uploaded file to another location using shutil.move. I first thought this was the problem, but it's the same when i use the included LocalUploadBackend.