Package: erudit-django-adv-cache-tag
django-adv-cache-tagDjango advanced cache template tag:versioningcompresspartial cachingeasily extendable/customizableReadable documentation onhttp://documentup.com/twidi/django-adv-cache-tagIntroductionFirst, notice that the arguments of the{% cache %}templatetag
provided bydjango-adv-cache-tagare the same as for the default
cache templatetag included in django, so it’s very easy to use this new
one.Withdjango-adv-cache-tagyou can :add a version number (int, string, date or whatever, it will be
stringified) to your templatetag : the version will be compared to
the cached one, and the exact same cache key will be used for the new
cached template, avoiding keeping old unused keys in your cache,
allowing you to cache forever.avoid to be afraid of an incompatible update in our algorithm,
because we also use an internal version number, updated only when the
internal algorithm changesdefine your own cache keys (or simply, just add the primary key (or
what you want, it’s a templatetag parameter) to this cache keycompress the data to be cached, to reduce memory consumption in your
cache backend, and network latency (but it will use more time and cpu
to compress/decompress, your choice)choose which cache backend will be useddefine{% nocache%}...{%endnocache %}blocks inside your cached
template, that will only be rendered when asked (for these parts, the
content of the template is cached, not the rendered result)easily define your own algorithm, as we provide a single class (with
short methods) you can inherit from, and simply change options or
whatever behavior you want, and define your own tags for themuse a variable for the name of your cache fragmentInstallationdjango-adv-cache-tagis available on PyPI:pip install django-adv-cache-tagStarting at version1.0, we only support python 3.If you upgrade from version < 1, note that the internal version number has changed, so all
cache will be reset.If you want python 2 support, you must install by passing the versionpip install 'django-adv-cache-tag<1.0'Or you can find it on Github:https://github.com/twidi/django-adv-cache-tag(for python2 version:https://github.com/twidi/django-adv-cache-tag/tree/python2)When installed, just addadv_cache_tagto yourINSTALLED_APPSin
thesettings.pyfile of your django project.See examples in the next sections to see how it works (basically the
same way as the default django cache templatetag)FeaturesVersioningDescriptionWith the default django cache templatetag, you can add as many arguments
as you want, including a version, or date, and then the cache key will
change if this version change. So your cache is updated, as expected.But the older key is not deleted and if you have a long expire time, it
will stay there for a very long time, consuming your precious memory.django-adv-cache-tagprovides a way to avoid this, while still
regenerating the cache when needed. For this, when activated, we use the
last argument passed to your templatetag as a “version number”, and
remove it for the arguments used to generate the cache key.This version will be used in thecontentof the cached template,
instead of thekey, and when the cache exists and is loaded, the
cached version will be compared to the wanted one, and if the two match,
the cache is valid and returned, else it will be regenerated.So if you like the principle of a unique key for a given template for a
given object/user or whatever, be sure to always use the same arguments,
except the last one, and activate theADV_CACHE_VERSIONINGsetting.Note that we also manage an internal version number, which will always
be compared to the cached one. This internal version number is only
updated when the internal algorithm ofdjango-adv-cache-tagchanges.
But you can update it to invalidate all cached templates by adding aADV_CACHE_VERSIONto your settings (our internal version and the
value from this settings will be concatenated to get the internal
version really used)SettingsADV_CACHE_VERSIONING, default toFalseADV_CACHE_VERSION, default to""ExampleIn the following template, ifADV_CACHE_VERSIONINGis set to True,
the key will always be the same, based on the string
“myobj_main_template” and the value ofobj.pk, but the cached
value will be regenerated each time theobj.date_last_updatedwill
change.So we set aexpire_timeof0, to always keep the template
cached, because we now we won’t have many copies (old ones and current
one) of it.The value to set to have no expiry may depend of your cache backend (it’s not always0).{%loadadv_cache%}{%cache0myobj_main_templateobj.pkobj.date_last_updated%}{{obj}}{%endcache%}Primary keyDescriptionIn the default django cache templatetag, the cache keys are like this
one:1:template.cache.your_fragment_name.64223ccf70bbb65a3a4aceac37e21016You may want to have more explicit cache keys, so withdjango-adv-cache-tagyou can add a “primary key” that will be added
between the fragment name and the hash:1:template.cache.your_fragment_name.your_pk.64223ccf70bbb65a3a4aceac37e21016Although the main use of this primary key is to have one cached fragment
per object, so we can use the object primary key, you can use whatever
you want, an id, a string…To add a primary key, simply set theADV_CACHE_INCLUDE_PKsetting toTrue, and the first argument (after the fragment’s name) will be
used as a pk.If you want this only for a part of your cache templatetags, read theExtending the default cache tagpart later in this readme (it’s
easy, really).Unlike the version, the primary key will be kept as an argument to
generate the cache key hash.SettingsADV_CACHE_INCLUDE_PK, default toFalseExampleA common use ofdjango-adv-cache-tagis to only use a primary key
and a version:{%cache0myobj_main_templateobj.pkobj.date_last_updated%}CompressionDescriptionThe default django cache templatetag simply saves the generated html in
the cache. Depending of your template, if may be a lot of html and your
cache memory will grow very quickly. Not to mention that we can have a
lot of spaces because of indentation in templates (two ways i know to
remove them withoutdjango-adv-cache-tag: the{% spaceless %}templatetag, provided by django, anddjango-template-preprocessor).django-adv-cache-tagcan do this for you. It is able to remove
duplicate spaces (including newlines, tabs) by replacing them by a
simple space (to keep the space behavior in html), and to compress the
html to be cached, via thezlib(andpickle) module.Of course, this cost some time and CPU cycles, but you can save a lot of
memory in your cache backend, and a lot of bandwidth, especially if your
backend is on a distant place. I haven’t done any test for this, but for
some templates, the saved data can be reduced from 2 ko to less than
one.To activate these feature, simply set toTrueone or both of the
settings defined below.WARNING : If the cache backend used use pickle and its default protocol,
compression is useless because binary is not really well handled and the
final size stored in the cache will be largely bigger than the
compressed one. So check for this before activating this option. It’s ok
for the default django backends (at least in 1.4), but not for
django-redis-cache, waiting for my pull-request, but you can check my
own version:https://github.com/twidi/django-redis-cache/tree/pickle_versionSettingsADV_CACHE_COMPRESS, default toFalse, to activate the
compression viazlibADV_CACHE_COMPRESS_LEVEL, default to-1, to set le compression level
forzlib(actually the default iszlib.Z_DEFAULT_COMPRESSION, which is-1, that will be in fact6as the actual default defined inzlib)ADV_CACHE_COMPRESS_SPACES, default toFalse, to activate the
reduction of blank characters.ExampleNo example since you don’t have to change anything to your templatetag
call to use this, just set the settings.Choose your cache backendDescriptionIn django, you can define many cache backends. But with the default
cache templatetag, you cannot say which one use, it will automatically
be the default one.django-adv-cache-tagcan do this for your by providing a setting,ADV_CACHE_BACKENDwhich will take the name of a cache backend
defined in your settings. And by extending the providedCacheTagobject, you can even define many backends to be used by many
templatetags, say one for heavily accessed templates, one for the
others… as you want. Read theExtending the default cache tagpart
to know more about this (it’s easy, really, but i already told you…)SettingsADV_CACHE_BACKEND, default to “default”ExampleNo example since, like for the compression, you don’t have to change
anything to your templatetag to use this, just set the setting.Partial cachingWith the default django cache templatetag, your templates are cached and
you can’t update them before display, so you can’t cache big parts of
html with a little dynamic fragment in it, for the user name, the
current date or whatever. You can cheat and save two templates
surrounding your dynamic part, but you will have more accesses to your
cache backend.django-adv-cache-tagallow the use of one or many{% nocache %}blocks (closed by{% endnocache %}) to put in your{% cache %}blocks. These{% nocache %}block will be saved “as is” in the
cache, while the rest of the block will be rendered to html. It’s only
when the template is finally displayed that the no-cached parts will be
rendered.You can have as many of these blocks you want.SettingsThere is no settings for this feature, which is automatically activated.Example{%cache0myobj_main_templateobj.pkobj.date_last_updated%}<p>This is the cached part of the template for{{obj}}, evaluated at{%now"r"%}.</p>{%nocache%}<p>This part will be evaluated each time :{%now"r"%}</p>{%endnocache%}<p>This is another cached part</p>{%endcache%}The fragment nameDescriptionThe fragment name is the name to use as a base to create the cache key, and is defined just
after the expiry time.The Django documentation statesThe name will be taken as is, do not use a variable.Indjango-adv-cache-tag, by settingADV_CACHE_RESOLVE_NAMEtoTrue, a fragment name
that is not quoted will be resolved as a variable that should be in the context.SettingsADV_CACHE_RESOLVE_NAME, default toFalseExampleWithADV_CACHE_RESOLVE_NAMEset toTrue, you can do this if you have a variable namedfragment_namein your context:{%cache0fragment_nameobj.pkobj.date_last_updated%}And if you want to pass a name, you have to surround it by quotes:{%cache0"myobj_main_template"obj.pkobj.date_last_updated%}WithADV_CACHE_RESOLVE_NAMEset toFalse, the default, the name is always seen as a string,
but if surrounded by quotes, they are removed.In the following example, you see double-quotes, but it would be the same with single quotes, or
no quotes at all:{%cache0"myobj_main_template"obj.pkobj.date_last_updated%}Extending the default cache tagIf the five settings explained in the previous sections are not enough
for you, or if you want to have a templatetag with a different behavior
as the default provided ones, you will be happy to know thatdjango-adv-cache-tagwas written with easily extending in mind.It provides a class,CacheTag(inadv_cache_tag.tag), which has
a lot of short and simple methods, and even aMetaclass (idea
stolen from the django models :D ). So it’s easy to override a simple
part.All options defined in theMetaclass are accessible in the class
viaself.options.some_fieldBelow we will show many ways of extending this class.Basic overrideImagine you don’t want to change the default settings (all toFalse,
and using thedefaultbackend) but want a templatetag with
versioning activated :Create a new templatetag file (myapp/templatetags/my_cache_tags.py)
with this:fromadv_cache_tag.tagimportCacheTagclassVersionedCacheTag(CacheTag):classMeta(CacheTag.Meta):versioning=Truefromdjangoimporttemplateregister=template.Library()VersionedCacheTag.register(register,'ver_cache')With these simple lines, you now have a new templatetag to use when you
want versioning:{%loadmy_cache_tags%}{%ver_cache0myobj_main_templateobj.pkobj.date_last_updated%}obj{%endver_cache%}As you see, just replace{% load adv_cache %}(or the django default{% load cache %}) by{% load my_cache_tags %}(your templatetag
module), and the{% cache %}templatetag by your new defined one,{% ver_cache ... %}. Don’t forget to replace the closing tag too:{% endver_cache %}. But the{% nocache %}will stay the same,
except if you want a new one. For this, just add a parameter to theregistermethod:MyCacheTag.register(register,'ver_cache','ver_nocache'){%ver_cache...%}cached{%ver_nocache%}not cached{%endver_nocache%}{%endver_cache%}Note that you can keep the namecachefor your tag if you know that
you will not load in your template another templatetag module providing
acachetag. To do so, the simplest way is:MyCacheTag.register(register)# 'cache' and 'nocache' are the default valuesAll thedjango-adv-cache-tagsettings have a matching variable in
theMetaclass, so you can override one or many of them in your own
classes. See the “Settings” part to see them.Internal versionWhen your template file is updated, the only way to invalidate all
cached versions of this template is to update the fragment name or the
arguments passed to the templatetag.Withdjango-adv-cache-tagyou can do this with versioning, by
managing your own version as the last argument to the templatetag. But
if you want to use the power of the versioning system ofdjango-adv-cache-tag, it can be too verbose:{%loadadv_cache%}{%withtemplate_version=obj.date_last_updated|stringformat:"s"|add:"v1"%}{%cache0myobj_main_templateobj.pktemplate_version%}...{%endcache%}{%endwith%}django-adv-cache-tagprovides a way to do this easily, with theADV_CACHE_VERSIONsetting. But by updating it,allcached
versions will be invalidated, not only those you updated.To do this, simply create your own tag with a specific internal version:classMyCacheTag(CacheTag):classMeta(CacheTag.Meta):internal_version="v1"MyCacheTag.register('my_cache')And then in your template, you can simply do{%loadmy_cache_tags%}{%my_cache0myobj_main_templateobj.pkobj.date_last_updated%}...{%endmy_cache%}Each time you update the content of your template and want invalidation,
simply change theinternal_versionin yourMyCacheTagclass (or
you can use a settings for this).Change the cache backendIf you want to change the cache backend for one templatetag, it’s easy:classMyCacheTag(CacheTag):classMeta:cache_backend='templates'But you can also to this by overriding a method:fromdjango.core.cacheimportget_cacheclassMyCacheTag(CacheTag):defget_cache_object(self):returnget_cache('templates')And if you want a cache backend for old objects, and another, faster,
for recent ones:fromdjango.core.cacheimportget_cacheclassMyCacheTag(CacheTag):classMeta:cache_backend='fast_templates'defget_cache_object(self):cache_backend=self.options.cache_backendifself.get_pk()<1000:cache_backend='slow_templates'returnget_cache(cache_backend)The value returned by theget_cache_objectshould be a cache backend
object, but as we only use thesetandgetmethods on this
object, it can be what you want if it provides these two methods. And
even more, you can override thecache_setandcache_getmethods
of theCacheTagclass if you don’t want to use the defaultsetandgetmethods of the cache backend object.Note that we also support the django way of changing the cache backend in the template-tag, using
theusingargument, to be set at the last parameter (without any space betweenusingand the
name of the cache backend).{%cache0myobj_main_templateobj.pkobj.date_last_updatedusing=foo%}Change the cache keyTheCacheTagclass provides three classes to create the cache key:get_base_cache_key, which returns a formatable string
(“template.%(nodename)s.%(name)s.%(pk)s.%(hash)s” by default ifinclude_pkisTrueor
“template.%(nodename)s.%(name)s.%(hash)s” ifFalseget_cache_key_args, which returns the arguments to use in the
previous stringget_cache_key, which combine the twoThe arguments are:nodenameparameter is the name of thetemplatetag: it’s
“my_cache” in{% my_cache ... %}nameis the “fragment name” of your templatetag, the value after
the expire-timepkis used only ifself.options.include_pkisTrue, and
is returned bythis.get_pk()hashis the hash of all arguments after the fragment name,
excluding the last one which is the version number (this exclusion
occurs only ifself.options.versioningisTrue)If you want to remove the “template.” part at the start of the cache key
(useless if you have a cache backend dedicated to template caching), you
can do this:classMyCacheTag(CacheTag):defget_base_cache_key(self):cache_key=super(MyCacheTag,self).get_base_cache_key()returncache_key[len('template:'):]# or [9:]Add an argument to the templatetagBy default, the templatetag provided byCacheTagtakes the same
arguments as the default django cache templatetag.If you want to add an argument, it’s easy as the class provides aget_template_node_argumentsmethod, which will work as for normal
django templatetags, taking a list of tokens, and returning ones that
will be passed to the real templatetag, aNodeclass tied to theCacheTag.Say you want to add afooargument between the expire time and the
fragment name:fromdjangoimporttemplatefromadv_cache_tag.tagimportCacheTag,NodeclassMyNode(Node):def__init__(self,nodename,nodelist,expire_time,foo,fragment_name,vary_on):""" Save the foo variable in the node (not resolved yet) """super(MyNode,self).__init__(self,nodename,nodelist,expire_time,fragment_name,vary_on)self.foo=fooclassMyCacheTag(CacheTag):Node=MyNodedefprepare_params(self):""" Resolve the foo variable to it's real content """super(MyCacheTag,self).prepare_params()self.foo=template.Variable(self.node.foo).resolve(self.context)@classmethoddefget_template_node_arguments(cls,tokens):""" Check validity of tokens and return them as ready to be passed to the Node class """iflen(tokens)<4:raisetemplate.TemplateSyntaxError(u"'%r' tag requires at least 3 arguments."%tokens[0])return(tokens[1],tokens[2],tokens[3],tokens[4:])Prepare caching of templatesThis one is not about overriding the class, but it can be useful. When
an object is updated, it can be better to regenerate the cached template
at this moment rather than when we need to display it.It’s easy. You can do this by catching thepost_savesignal of your
model, or just by overriding itssavemethod. For this example we
will use this last solution.The only special thing is to know the path of the template where your
templatetag is. In my case, i have a template just for this (included in
other ones for general use), so it’s easier to find it and regenerate it
as in this example.As we are not in a request, we don’t have theRequestobject here,
so context processors are not working, we must create a context object
that will be used to render the template, with all needed variables.fromdjango.templateimportloader,ContextclassMyModel(models.Model):# your fieldsdefsave(self,*args,**kwargs):super(MyModel,self.save(*args,**kwargs)template='path/to/my_template_file_with_my_cache_block.html'context=Context({'obj':self,# as you have no request, we have to add stuff from context processors manually if we need them'STATIC_URL':settings.STATIC_URL,# the line below indicates that we force regenerating the cache, even if it exists'__regenerate__':True,# the line below indicates if we only want html, without parsing the nocache parts'__partial__':True,})loader.get_template(template).render(context)Load data from database before renderingThis is a special case. Say you want to display a list of objects but
you have only ids and versions retrieved from redis (withZSET, with
id as value and updated date (which is used as a version) as score , for
example)If you know you always have a valid version of your template in cache,
because they are regenerated every time they are saved, as seen above,
it’s fine, just add the object’s primary key as thepkin your
templatetag arguments, and the cached template will be loaded.But if it’s not the case, you will have a problem: when django will
render the template, the only part of the object present in the context
is the primary key, so if you need the name or whatever field to render
the cached template, it won’t work.Withdjango-adv-cache-tagit’s easy to resolve this, as we can load
the object from the database and adding it to the context.Viewdefmy_view(request):objects=[dict(pk=val[0],date_last_updated=val[1])forvalinredis.zrevrange('my_objects',0,19,withscores=True)]returnrender(request,"my_results.html",dict(objects=objects))Template “my_results.html”{%forobjinobjects%}{%include"my_result.html"%}{%endfor%}Template “my_result.html”{%loadmy_cache_tags%}{%my_cache0myobj_main_templateobj.pkobj.date_last_updated%}{{obj}}{%endmy_cache%}TemplatetagInmyapp/templatetags/my_cache_tagsfrommy_app.modelsimportMyModelclassMyCacheTag(CacheTag):classMeta(CacheTag.Meta):""" Force options """include_pk=Trueversioning=Truedefcreate_content(self):""" If the object in context is not a real model, load it from db """ifnotisinstance(context['obj'],MyObject):context['obj']=MyModel.objects.get(id=self.get_pk())super(MyCacheTag,self).create_content()MyCacheTag.register('my_cache')Careful with this, it generates as database requests as objects to be
loaded.And more…If you want to do more, feel free to look at the source code of theCacheTagclass (intag.py), all methods are documented.Settingsdjango-adv-cache-tagprovide 5 settings you can change. Here is the
list, with descriptions, default values, and corresponding fields in theMetaclass (accessible viaself.options.some_fieldin theCacheTagobject)ADV_CACHE_VERSIONINGto activate versioning, default toFalse(versioningin theMetaclass)ADV_CACHE_COMPRESSto activate compression, default toFalse(compressin theMetaclass)ADV_CACHE_COMPRESS_LEVELto set the compression level (from1(min
compression) to9(max compression), default to-1(equivalent to6) (compress_levelin theMetaclass)ADV_CACHE_COMPRESS_SPACESto activate spaces compression, default
toFalse(compress_spacesin theMetaclass)ADV_CACHE_INCLUDE_PKto activate the “primary key” feature,
default toFalse(include_pkin theMetaclass)ADV_CACHE_BACKENDto choose the cache backend to use, default to"default"(cache_backendin theMetaclass)ADV_CACHE_VERSIONto create your own internal version (will be
concatenated to the real internal version ofdjango-adv-cache-tag), default to""(internal_versionin
theMetaclass)How it worksHere is a quick overview on how things work indjango-adv-cache-tagPartial cachingYour template :{%loadadv_cache%}{%cache...%}foo{%nocache%}bar{%endnocache%}baz{%endcache%}Cached version (we ignore versioning and compress here, just to see how
it works):foo{%endRAW_xyz%}bar{%RAW_xyz%}bazWhen cached version is loaded, we parse :{%RAW_xyz%}foo{%endRAW_xyz%}bar{%RAW_xyz%}baz{%endRAW_xyz%}The first{% RAW_xyz %}and the last{% endRAW_xyz %}are not
included in the cached version and added before parsing, only to save
some bytes.Parts between{% RAW_xyz %}and{% endRAW_xyz %}are not parsed
at all (seen as aTextNodeby django)Thexyzpart of theRAWandendRAWtemplatetags depends on
theSECRET_KEYand so is unique for a given site.It allows to avoid at max the possible collisions with parsed content in
the cached version.We could have used{% nocache %}and{% endnocache %}instead of{% RAW_xyz %}and{% endRAW_xyz %}but in the parsed template,
stored in the cache, if the html includes one of these strings, our
final template would be broken, so we use long ones with a hash (but we
can not be sure at 100% these strings could not be in the cached html,
but for common usages it should suffice)Licensedjango-adv-cache-tagis published under the MIT License (see the
LICENSE file)Running testsIfadv_cache_tagis in theINSTALLED_APPSof your project, simply
run:django-admin test adv_cache_tag(you may want to usedjango-adminor./manage.pydepending on
your installation)If you are in a fresh virtualenv to work onadv_cache_tag, install
the django version you want:pip install djangoThen make theadv_cache_tagmodule available in your python path.
For example, withvirtualenv-wrapper, considering you are at the
root of thedjango-adv-cache-tagrepository, simply do:add2virtualenv .Or simply:pip install -e .Then to run the tests, this library provides a test project, so you can
launch them this way:DJANGO_SETTINGS_MODULE=adv_cache_tag.tests.testproject.settings django-admin.py test adv_cache_tagOr simply launch theruntests.shscript (it will run this exact
command):./runtests.shSupported versionsDjango versionPython versionLibrary version1.7 to 1.112.70.41.73.41.1.31.8 to 1.103.4, 3.51.1.31.113.4 to 3.61.1.32.03.4, to 3.71.1.32.13.5 to 3.71.1.32.23.5 to 3.81.1.33.03.6 to 3.81.1.3Support for Python 2 is dropped since version 1 ofdjango-adv-cache-tag