Package: django-dumper
django-dumperprovides full site caching, similar to Django’s,
along with path based invalidation based on model saves.
It will cache every page that is not inDUMPER_PATH_IGNORE_REGEXforever. So for it to be effective, all of your pages must
be invalidated when neccesary, by specifying in your models
which paths should be invalidated those models are saved.Why…?django-dumperwas created to scratch a rather specific itch. I was having
trouble reducing load times on pages where there were lots of images. By
default, if you want to render an image’s url, width, and height, in a template then
that hits the storage backend three times per image. With a remote backend,
like S3, this creates long and unreliable page load times. If you are smarter
and createcached height and width fields, then you can reduce this to one
hit. This is still not ideal for a page with 100+ images. So I thought, the only
time these images ever change, is when a model saves. And then I started
thinking, actually the only time any of my pages changed was when a model
saved. Of course, I still wanted my page load times to be as low as possible
before caching, but why would i re-render these pages on every request, if
they would be identical for every visitor, until someone changed a model?So I set about to build an app that would allow me to do just that. It
would cache the full content of each response indefinitely. It would then
invalidate certain responses, based on their paths, whenever a model was saved.
For instance, if the/ice-cream/page displays links to each flavor and/ice-cream/<flavor-name>/has detailed information on a flavor, then
every time a flavor is saved, it should not only invalidate its specific detail
page, but also the general list page. This is definitely a brute force approach,
but it makes sense to me because it issafe. You might over invalidate, but,
if setup correctly, you will never have stale caches.This is by no means an all purpose caching app. Every page rendered by your site
must be determined only by models. Detail and list views are examples of pages
determined by models. Also, if your site differentiates at based on request
headers (cookies, languages, etc…) then this will not work, because it will
serve the same version to all visitors.InstallationInstallation is as easy as:pip install django-dumperSetupConfiguration is similar to Django’sper sitecache.You’ll need to add'dumper.middleware.cache.UpdateCacheMiddleware'and'dumper.middleware.cache.FetchFromCacheMiddleware'to yourMIDDLEWARE_CLASSES setting, as in this example:MIDDLEWARE_CLASSES=('dumper.middleware.UpdateCacheMiddleware','django.middleware.common.CommonMiddleware','dumper.middleware.FetchFromCacheMiddleware',)Then, add the following optional settings to your Django settings file:DUMPER_CACHE_ALIAS– The cache alias to use for storage. Defaults'default'DUMPER_KEY_PREFIX– If the cache is shared across multiple sites
using the same Django installation, set this to the name of the site,
or some other string that is unique to this Django instance, to
prevent key collisions. Defaults to'dumper.cached_path.'.DUMPER_PATH_IGNORE_REGEX– If matched on the path, then
these pages will not be cached. By default it won’t cached the admin^/admin/HintIf you use Django Grappelli, then you shouldn’t be caching
any paths under/grappelli/and if you are serving media or static from
your app, you should ignore those as well.DUMPER_PATH_IGNORE_REGEX=r'^/(?:(?:admin)|(?:grappelli)|(?:media))/'UsageTo invalidate certain paths on a model save and delete, register that model
usingdumper.register. It will invalidate every path returned by thedependent_pathsmethod.fromdjango.dbimportmodelsimportdumperclassIceCream(models.Model):slug=models.CharField(max_length=200)defget_absolute_url(self):return'/'+self.slugdefdependent_paths(self):'''Returns a list of paths to invalidate when this model is updated'''return[self.get_absolute_url()]dumper.register(IceCream)dependent_pathscan also returns the paths of related objects to invalidate
them as well. For instance if eachIceCreamhad some relatedSizesthen if one of those sizes is modified, that should invalidate theIceCreamas well.fromdjango.dbimportmodelsimportdumperclassIceCream(models.Model):slug=models.CharField(max_length=200)sizes=models.ManyToManyField(Size,related_name='ice_creams')defget_absolute_url(self):return'/'+self.slugdefdependent_paths(self):'''Returns a list of paths to invalidate when this model is updated'''return[self.get_absolute_url()]classSize(models.Model):slug=models.CharField(max_length=200)defget_absolute_url(self):return'/'+self.slugdefdependent_paths(self):forice_creaminself.ice_creams:yieldice_cream.get_absolute_url()yieldself.get_absolute_url()dumper.register(IceCream)dumper.register(Size)DebuggingThedumperpackage hasDEBUGlogging in place for the midleware
and for the invalidation. To enable this, just make sure that
any logs coming fromdumperwith the levelDEBUGare shown.The simplest way to do this would be to this in yoursettings.pyLOGGING={'version':1,'disable_existing_loggers':False,'formatters':{'name':{'format':'%(name)s:%(message)s'},},'handlers':{'console':{'level':'DEBUG','class':'logging.StreamHandler','formatter':'name'},},'loggers':{'dumper':{'level':'DEBUG','handlers':['console',]}}}AdviceI would recommend enablingETags. That way the whole response
won’t have to be sent to the user, only the header, if the ETAG is the same.The Django document ion does not cohesively describe how your middleware
should be ordered, howeverthis stack overflowdiscussion does a fine job.InternalsCache Middlewaredumper/middleware.pyMy caching is based off of Django’sper site cache, but much simpler.
Originally I just used their cache, but this greatly complicated my code
and made it harder to understand. This is because their cachecreates different cached versions. for the same URL based on theVaryHTML header.
It is much more complicated to implement path based invalidation, if other things
besides the path are being use to generate the cache key. For instance, when I was
supporting the Django middleware I had to figure out a way to delete every cached
version of the path.If your pages do vary based on anything besides the path and HTTP method,
then you should not cache them withdjango-dumper. Either ignore them
with theDUMPER_PATH_IGNORE_REGEXsetting or don’t use the project at all
if all of your pages fall under this category.Invalidate Pathsdumper/invalidation.pyIn order to invalidate a model when it saves, we get the path’s that should
be invalidated from the model, and then remove the cache keys that correspond
to those paths. Each cache key is made up of a path plus a HTTP method.Invalidating on Model Saves:dumper/site.pyWhen you register a model a invalidation function to three signals.
That function gets the paths from the model and then usesdumper/invalidation.pyto delete them. The three signals it registers with arepost_save,pre_delete,
andm2m_changed. The last signal is called whenever any member that relationship
is added, deleted, or changed. It most likely calls the
invalidation function more than once if a many to many relationship is changed,
but is harmless, besides the slight performance hit from hitting the cache backend.ContributingIf you find issues or would like to see a feature suppored, head over to
theissues sectionand report it. Don’t be agraid, go ahead, do it!To contribute code in any form, fork the repository and clone it locally.
Create a new branch for your feature:git commit -b feature/whatever-you-likeThen make sure all the tests past (and write new ones for any new features)With Docker Compose and Docker:docker-compose run tests
# run `docker-compose build` if you change the required packages before testing againNormally:pip install -e .
pip install -r requirements-dev.txt
django-admin.py test --settings=test.settingsCheck if the README.rst looks right:restview --long-descriptionThen push the finished feature to github and open a pull request form the branch.New ReleaseTo create a new release:Add changes toCHANGES.txtChange version insetup.pypython setup.py registerpython setup.py sdist upload