Package: django-cached-field
IntroductionUsing Django ORM and Celery, cache expensive-to-calculate attributes.ExampleSay you have a CTO who believes everything belongs in a relational database and
a slow method on one of your models:classLamppost(models.Model):# ...@propertydefslow_full_name(self):ackermann(5,2)return'The%s%sof%s'%(self.weight,self.first_name,self.country)Ugh; too slow. Let’s cache that (but not with, say, a dedicated caching system).
We’ll want a few tools.Celerywithdjango-celerywill need to be set up
and humming along smoothly. Then we’ll add in our cached field and rename our
method appropriately:fromdjango_cached_fieldimportCachedIntegerFieldclassLamppost(models.Model):# ...slow_full_name=CachedTextField(null=True)defcalculate_slow_full_name(self):ackermann(5,2)return'The%s%sof%s'%(self.weight,self.first_name,self.country)(Yeah,calculate_*is just a convention. I clearly haven’t given
up the ruby ghost, but you can pass in your own method name with
thecalculation_method_namearg to the field declaration.)Next, migrate your db schema to automatically include the new cache
control fields. Note that at least two fields will be added
to this table,cached_slow_full_nameof typetext,slow_full_name_recalculation_neededof typeboolean, probably
defaulting to true, and possiblyslow_full_name_expires_afterof
typedatetime, if we passtemporal_triggers=Trueinto the field
declaration (more on that later).Already that’s kinda better.lamppost.slow_full_namemay take a
while the first time it gets called for a given record, but from then
on, it’ll be nigh instant. Of course, at this point, it will never
change after that first call.The remaining important piece of the puzzle is to invalidate our cache
usingflag_slow_full_name_as_stale. It is probably changed in some
views.py (this example code could be more clever about noticing if the
relevant values are updated):@render_to('lamppost/edit.html')defedit(request,lamppost_id):lamppost=Lamppost.objects.get(pk=lamppost_id)form=LamppostForm(request.POST,lamppost)ifform.is_valid():form.save()lamppost.flag_slow_full_name_as_stale()return{'form':form,'lamppost':lamppost}This is the hardest part as the developer.Caching requires you
hunt down every place the value could be changed and calling thatflag_slow_full_name_as_stalemethod. Is country assigned a random
new value every morning at cron’o’clock? That flag had best be stale
by cron’o’one. Do you calculate weight based on the sum of all
associated pigeons? Hook into the pigeons landing. And takeoff. And
everything that changes an individual pigeon’s weight. As Francis
Bacon said, “There are only two hard problems in programming:
naming, cache invalidation and off-by-one errors.”One possible invalidation scheme you might want to use is expiration
dates. We know the pigeons on our lamppost are going to die and turn
into ghosts, right:classPigeon(models.Model):death_date=models.DateField()defdie(self):self.weight=0self.save()And rather than bother the pigeon-death-handling system, we’ll take
note of their death as they land:classLamppost(models.Model):#...defnotice_pigeon_landing(self,pigeon):earliest=self.pigeon_set.all().aggregate(models.Min('death_date'))['death_date']self.expire_slow_full_name_after(earliest)Or maybe you only want the cache to ever be valid for 30 minutes, lestTheyhave too easy a job tracking your thoughts.So, yeah, you get the idea.InstallationYou can make things easy on yourself:pipinstalldjango-cached-fieldOr, for a more artisanal feeling, you can clone the repo and install it
using python and setup.py:gitclonegit@github.com:outofculture/django-cached-field.gitcddjango-cached-field/pythonsetup.pyinstallTested with minimum versions python 2.7, django 1.3.1, celery 2.3.1,
and django-celery 2.3.3. Should be compatible with as recent as python
3.8, django 3.0.5, celery 3.1.26.post2 and django-celery 3.3.1.ConfigurationIf you’re going to have expiration dates on your values, and you want
to use timezone-aware datetimes, you will need to set this setting to
True:CACHED_FIELD_USE_TIMEZONE=False# defaultOne setting for test environments: whether recalculation should happen
when flagged as stale (default) or be left to the next time the
attribute is accessed. This is useful for optimizing testing
environments where you don’t care that your cached values are invalid
or that the expense of calculation is applied to a user. Note that, in
this situation, you wouldn’t need celery. :CACHED_FIELD_EAGER_RECALCULATION=True# defaultThis is a global option, so individual exceptions should instead be
handled by passing theand_recalculateargument to theflag_FIELD_as_stalecall.Configuration for Django < 1.6Use of this library under at least version >= 1.6 of Django should not
require any configuration changes; just import and use. For older
Djangos, two settings changes are probably required for things to
work: make sure it’s a registered app, make sure celery sees its tasks
file:INSTALLED_APPS+=['django_cached_field',]CELERY_IMPORTS+=['django_cached_field.tasks',]CaveatsAll ORM methods (e.g.order_by,filter) can only access this field throughcached_FIELD.recalculate_FIELDuses.update(cached_FIELD=to set the value. Don’t expect.saveto be called.flag_FIELD_as_staleuses.update, as well.This may break if you try to add this mixin to a field class that multiply-inherits (I’m currently grabbing an arbitrary, non-CachedFieldMixin class and making the real field with it).The FIELD_recalculation_needed field is accessed by regex in at least one place, so problems will result from user fields that match the same pattern.Exceptions in a post-hook will potentially break saves of the data.TODOFigure out if we can turn temporal_triggers into a celery job that happens once at the given time.All my tests are in the project I pulled this out of, but based on models therein. I don’t have experience making tests for standalone django libraries. Someone wanna point me to a tutorial?Recalculation task will not adapt to recalculation_needed_field_name optionReplace use of _recalculation_needed regex with class-level registry of cached fields.