richleland/django-cumulus — pull request #70 by rizumu (2013-02-02 22:51:44+00:00)
Title: rewrite staticfiles/management command to use swiftclient

Here I have ported the management commands (except for container_delete) and the staticfiles portion of this app to swiftclient and am currently working on the storage backend.

Swiftclient is definitely the way to go because I have had zero timeout errors which has been the showstopping problem with python-cloudfiles / django-cumulus.

I read the the goal is to move cumulus to django-storages, so I hope these changes are in line with the plans for that. https://bitbucket.org/david/django-storages/issue/141/add-support-for-openstack-swift

A few important differences/caveats:

1. The syncstatic --include and --exclude are now more robust and use regular expression matching, but as a result using collectstatic (STATICFILES_STORAGE) to sync is not a good option because it it automatically syncs and therefore isn't smart enough to be able to include and exclude. Maybe there is a pre/post collect hook in django.contrib.staticfiles that would allow this, but I looked and couldn't find it.

2. I got stuck on the multiprocessing logic in the container_delete management command. I converted most of it to use swiftclient, but put it aside deleting files in parallel to working on the storage backend. In fact most of the need for multiprocessing is eliminated because swiftclient returns all filenames as json, where as cloudfiles required a request per file. Still, it would be nice to do deletion in parallel.

3. I've tried to use double quotes consistently, so sorry for some of the git diff noise when switching from single to double.

Ok, I'll be working on the storage backend and tests this coming week. Maybe it is a good idea to finish the port to python-swiftclient before merging back to django-storages. It really is a large enough improvement over python-cloudfiles to merit the wait.