Package: dogpile-backend-redis-advanced
This package supports Python2 and Python3This package DOES NOT supportdogpile.cache>=1.0. Support is planned, but there have been several major API changes that are incompatible.dogpile_backend_redis_advancedThis is a plugin for thedogpile.cachesystem that offers some alternatives
to the standardRedisdatastore implementation.Two new backends are offered:backenddescriptiondogpile_backend_redis_advancedextends thedogpile.cache.redisbackend and allows for custom pickling overridesdogpile_backend_redis_advanced_hstoreextendsdogpile_backend_redis_advancedand allows for some specific hstore operationsThere is a negligible performance hit indogpile_backend_redis_advanced_hstore,
as cache keys must be inspected to determine if they are an hstore or not -- and
there are some operations involved to coordinate values.Additionally, some behavior is changed:The constructor now accepts alock_classargument, which can be used to
wrap a mutex and alter how releases are handled.  This can be necessary if you
have a distributed lock and timeout or flush issues (via LRU or otherwise).
A lock disappearing in Redis will raise a fatal exception under the standard
Redis backend.The constructor now accepts alock_prefixargument, which can be used to
alter the prefix used for locks.  The standard Redis backend uses_lockas
the prefix -- which can be hard to read or isolate for tests.  One might want
to use "_" as the lock prefix (so thatkeys "\_*"will show all locks).Purpose:Mike Bayer'sdogpile.cacheis an excellent package for general purpose
development.The system offers 3 key features:Elegant read-through caching functionality.A locking mechanism that ensures only the first request of a cache-miss will
create the resource (turning the rest into consumers of the first-requestor's
creation).Integrated cache expiry against time and library versions.Unfortunately, the integrated cache expiry feature comes at a cost -- objects
are wrapped into a tuple with some metadata and pickled before hitting the
datastore.The additional metadata or pickle format may not be needed or wanted.  Look how
the size of "a" grows by the time it becomes something passed off to Redis:typeexamplestringapickle(string)S'a'\np0\n.CachedValue(string)('a', {'ct': 1471113698.76127, 'v': 1})pickle(CachedValue(string) )cdogpile.cache.api\nCachedValue\np0\n(S'a'\np1\n(dp2\nS'ct'\np3\nF1471113698.76127\nsS'v'\np4\nI1\nstp5\nRp6\n.By adding in hooks for custom serializers, this backend lets developers choose
better ways to cache data.You may want a serializer that doesn't care about the expiry of cached data, so
just uses simpler strings.:typeexample 1example 2stringamellifluousjson.dumps(string)"a""mellifluous"msgpack.packb(string)\xa1a\xabmellifluousOr, you may want to fooldogpile.cacheby manipulating what the cached is.
Instead of using a Python dict, of time and API version, you might just track
the time but only to the second.typeexample 1example 2AltCachedValue(string)('a', 1471113698)('mellifluous', 1471113698)json.dumps(AltCachedValue(string))'["a", 1471113698]''["mellifluous", 1471113698]'msgpack.packb(AltCachedValue(string))'\x92\xa1a\xceW\xafi\xe2''\x92\xabmellifluous\xceW\xafi\xe2'This is howdogpile.cachestores "a":cdogpile.cache.api\nCachedValue\np0\n(S'a'\np1\n(dp2\nS'ct'\np3\nF1471113698.76127\nsS'v'\np4\nI1\nstp5\nRp6\n.This package lets us cache a raw string and trickdogpile.cacheinto
thinking our data parcel is "timely":aOr, we include a simpler version of the time, along with a different serializer.This packet of data and time:["a", 1471113698]Is then serialized to:\x92\xa1a\xceW\xafi\xe2If you cache lots of big objects,dogpile.cache's overhead is minimal -- but
if you have a cache that works for mapping short bits of text, like ids to
usernames (and vice-versa) you will see considerable savings.Another way to makeRedismore efficient is to use hash storage.Let's say you have a lot of keys that look like this:region.set("user-15|posts", x)
region.set("user-15|friends", y)
region.set("user-15|profile", z)
region.set("user-15|username", z1)You could makeRedisa bit more efficient by using hash storage, in which
you have 1 key with multiple fields:region.hset("user-15", {'posts': x,
						'friends', y,
						'profile', z,
						'username', z1,
						})Redis tends to operate much more efficiently in this situation (more below),
but you can also save some bytes by not repeating the key prefix. Instagram's
engineering team has a great article on thisInstagram Engineering.90% ofdogpile.cacheusers who chooseRediswill never need this
package.  A decent number of other users with large datasets have been trying to
squeeze every last bit of memory and performance out of their machines -- and
this package is designed to facilitate that.Usage:myfile.py# importing will register the plugins
import dogpile_backend_redis_advancedthen simply configuredogpile.cachewithdogpile_backend_redis_advancedordogpile_backend_redis_advanced_hstoreas the backend.RedisAdvancedBackendTwo new configuration options are offered to specify custom serializers vialoadsanddumps.  The default selection is to usedogpile.cache's choice
ofpickle.This option was designed to supportmsgpackas the serializer:import msgpack
from dogpile.cache.api import CachedValue

def msgpack_loads(value):
    """pickle maintained the `CachedValue` wrapper of the tuple
       msgpack does not, so it must be added back in.
       """
    value = msgpack.unpackb(value, use_list=False)
    return CachedValue(*value)

region = make_region().configure(
    arguments= {'loads': msgpack_loads,
                'dumps': msgpack.packb,
                }
    )One can also abuse/misusedogpile.cacheand defer all cache expiry toRedisusing this serializer hook.dogpile.cachedoesn't cache your value as-is, but wraps it in a CachedValue
object which contains an API version and a timestamp for the expiry.This format is necessary for most cache backends, butRedisoffers the
ability to handle expiry in the cloud.  By using the slim msgpack format and
only storing the payload, you can drastically cut down the bytes needed to store
this information.This approach SHOULD NOT BE USED by 99% of users.  However, if you do aggressive
caching, this will allow you to leveragedogpile.cache's excellent locking
mechanism for handling read-through caching while slimming down your cache size
and the traffic on-the-wire.import time
from dogpile.cache.api import CachedValue
from dogpile.cache.region import value_version
import msgpack

def raw_dumps(value):
    ''''pull the payload out of the CachedValue and serialize that
    '''
    value = value.payload
    value = msgpack.packb(value)
    return value

def raw_loads(value):
    ''''unpack the value and return a CachedValue with the current time
    '''
    value = msgpack.unpackb(value, use_list=False)
    return CachedValue(
        value,
        {
            "ct": time.time(),
            "v": value_version
        })

region = make_region().configure(
    arguments= {'loads': msgpack_loads,
                'dumps': msgpack.packb,
                'redis_expiration_time': 1,
                }
    )RedisAdvancedHstoreBackendThis backend extendsRedisAdvancedBackendwith drop-in support for Hash
storage under Redis.If key names are tuples, they will be treated as hash operations on Redis.By settingredis_expiration_time_hashto a boolean value, you can control
how expiry times work within RedisThis backend has a slight, negligible, overhead:All key operations (get/get_multi/set/set_multi/delete) require an
inspection of keys.get_multirequires the order of keys to be tracked, and results from
multipleget/hgetoperations are then correlated.set_multirequires the mapping to be analyzed and bucketed into different
hmsetsredis_expiration_time_hashallows some extended management of expiry in Redis.
By default it is set toNone.False- ignore hash expiry. (never set a TTL in Redis)None- setredis_expiration_timeon new hash creation only. This requires
a check to theRediskey before a set.True- unconditionally setredis_expiration_timeon every hash key
set/update.Please note the following:Redismanages the expiry of hashes on the key, making it global for all
fields in the hash.Redisdoes not support setting a TTL on hashes while doing another
operation.  TTL must be set via another request.Ifredis_expiration_time_hashis set toTrue, there will be 2 calls to
theRedisAPI for every key:hsetorhmsetthenexpires.Ifredis_expiration_time_hashis set toNone, there will be 2-3 calls to
theRedisAPI for every key:exists,hsetorhmset, and possiblyexpires.Memory Savings and Suggested UsageRedis is an in-memory datastore that offers persistence -- optimizing storage is
incredibly important because the entire set must be held in-memory.Example DemoThe attacheddemo.py(results indemo.txt) shows some potential approaches
to caching and hashing by priming aRedisdatastore with some possible
strategies of a single dataset.It's worth looking atdemo.txtto see how the different serializesr encode the
data -- sample keys are pulled for each format.testmemory bytesmemory humanrelativettl on Redis?ttl in dogpile?backendencoderregion_redis249399504237.85M0%YYdogpile.cache.redispickleregion_json222924496212.60M89.38%YYdogpile_backend_redis_advancedjsonregion_msgpack188472048179.74M75.57%YYdogpile_backend_redis_advancedmsgpackregion_redis_local181501200173.09M72.78%-Ydogpile.cache.redispickleregion_json_raw171554880163.61M68.79%Y-dogpile_backend_redis_advancedjsonregion_msgpack_raw170765872162.86M68.47%Y-dogpile_backend_redis_advancedmsgpackregion_json_local162612752155.08M65.20%-Ydogpile_backend_redis_advancedjsonregion_json_local_int128648576122.69M57.71%-Y,int(time)dogpile_backend_redis_advancedjsonregion_msgpack_local128160048122.22M51.39%-Ydogpile_backend_redis_advancedmsgpackregion_msgpack_local_int126938576121.06M50.89%-Y,int(time)dogpile_backend_redis_advancedmsgpackregion_json_raw_local111241280106.09M44.60%--dogpile_backend_redis_advancedjsonregion_msgpack_raw_local110455968105.34M44.29%--dogpile_backend_redis_advancedmsgpackregion_msgpack_raw_hash2851886427.20M11.44%Y, only keys-dogpile_backend_redis_advanced_hstoremsgpackregion_json_raw_hash2483616023.69M9.96%Y, only keys-dogpile_backend_redis_advanced_hstorejsonNotes:the_localvariants do not set a TTL on Redisthe_rawvariants strip out the dogpile CachedValue wrapper and only store
the payloadthe_msgpackvariants use msgpack instead of picklethe_jsonvariants use json instead of picklethe_intvariant applies int() to the dogpile timestamp, dropping a few
bytes per entryWait WHAT? LOOK ATregion_msgpack_raw_hashandregion_json_raw_hash- that's
a HUGE savings!Yes.The HSTORE has considerable savings due to 2 reasons:Redisinternally manages a hash much more effectively than keys.Rediswill only put an expiry on the keys (buckets), not the hash fieldsHSTORE ends up being a much tighter memory usage for this example set, as we're
setting 100 fields in each key.  The savings would not be so severe if you were
setting 5-10 fields per keyNote thatregion_msgpack_raw_localandregion_json_raw_localshould not be
used unless you're running a LRU -- they have no expiry.AssumptionsThis demo is assuming a few things that are not tested here (but there are
plenty of benchmarks on the internet showing this):msgpack is the fastest encoder for serializing and deserializing data.json outperforms cpickle on serializing; cpickle outperforms json on
deserializing data.Here are some benchmarks and links:https://gist.github.com/justinfx/3174062https://gist.github.com/cactus/4073643http://www.benfrederickson.com/dont-pickle-your-data/CaveatsIn the examples above, we deal with (de)serializing simple, native, datatypes:string,int,bool,list,dict,tuple.  For these datatypes, msgpack
is both the smallest datastore and the fastest performer.If you need to store more complex types, you will need to provide a custom
encoder/decoder and will likely suffer a performance hit on the speed of
(de)serialization.  Unfortunately, the more complex data types that require
custom encoding/decoding include standarddatetimeobjects, which can be
annoying.The filecustom_serializer.pyshows an example class for handling
(de)serialization --MsgpackSerializer.  Some commondatetimeformats are
supported; they are encoded as a specially formatted dict, and decoded
correspondingly.  A few tricks are used to shave off time and make it roughly
comparable to the speed of pickle.Key Takeawaysthis was surprising - while the differences are negligible on small datasets,
usingRedisto track expiry on long data-sets is generally not a good
idea(!).dogpile.cachetracks this data much more efficiently.  you can
enable an LRU policy inRedisto aid in expiry.msgpack and json are usually fairly comparable in size [remember the
assumption that msgpack is better for speed].reformatting thedogpile.cachemetadata (replacing adictanint()of
the expiry) saves a lot of space under JSON when you have small payloads. the
strings are a fraction of the size.msgpack is really good with nested data structuresThe following payloads for1are strings:region_json_local =        '[10, {"v": 1, "ct": 1471113698.76127}]'
region_json_local_int =    '[10, 1471113753]'
region_msgpack_local =     '\x92\n\x82\xa1v\x01\xa2ct\xcbA\xd5\xeb\x92\x83\xe9\x97\x9a'
region_msgpack_local_int = '\x92\n\xceW\xafct'So what should you use?There are several tradeoffs and concepts to consider:Do you want to access information outside ofdogpile.cache(in Python
scripts, or even in another language)Are you worried about the time to serialize/deserialize?  are you write-heavy
or read-heavy?Do you want the TTL to be handled byRedisor within Python?What are your expiry needs?  what do your keys look like?  there may not be
any savings possible.  but if you have a lot of recycled prefixes, there
could be.What do your values look like?  How many are there?This test uses a particular dataset, and differences are inherent to the types
of data and keys. Using the strategies from theregion_msgpack_raw_hashon
our production data has consistently dropped a 300MBRedisimprint to the
60-80MB range.TheRedisconfiguration file is also enclosed. The above tests are done withRediscompression turned on (which is why memory size fluctuates in the full
demo reporting).Custom Lock ClassesIf your Redis db gets flushed the lock will disappear. This will cause the Redis
backend to raise an exception EVEN THOUGH you have succeeded in generating your
data.By using alock_class, you can catch the exception and decide what to do --
log it?, continue on, raise an error?  It's up to you!import redis.exceptions

class RedisDistributedLockProxy(object):
	"""example lock wrapper
	this will silently pass if a LockError is encountered
	"""
	mutex = None

	def __init__(self, mutex):
		self.mutex = mutex

	def acquire(self, *_args, **_kwargs):
		return self.mutex.acquire(*_args, **_kwargs)

	def release(self):
		# defer imports until backend is used
		global redis
		import redis  # noqa
		try:
			self.mutex.release()
		except redis.exceptions.LockError, e:
			# log.debug("safe lock timeout")
			pass
		except Exception as e:
			raiseTo DoI've been experimenting with handling the TTL within a hash bucket (instead of
using theRedisordogpile.cachemethods). This looks promising.  The
rationale is that it is easier forRedisto get/set an extra field from the
same hash, than it is to do separate calls to TTL/EXPIRES.in code:- hset('example', 'foo', 'bar')
- expires('example', 3600)
+ hmset('example', {'foo': 'bar',
					'expires': time.time() + 3600,
					}I've also been experimenting with blessing the result into a subclass ofdictthat would do the object pair decoding lazily as-needed.
That would speed up most use cases.MaturityThis package is pre-release.  I've been using these strategies in production
via a custom fork ofdogpile.cachefor several years, but am currently
migrating it to a plugin.Maintenance and Upstream CompatibilitySome files in /tests are entirely fromdogpile.cacheas-is:/tests/conftest.py/tests/cache/_init_.py/tests/cache/_fixtures.pyThey are versions fromdogpile.cache0.6.2The core file,/cache/backends/redis_advanced.pyinherits fromdogpile.cache's/cache/backends/redis.pyTestingThis ships with full tests.Much of the core package and test fixtures are fromdogpile.cacheand
copyright from that project, which is available under the MIT license.Tests are handled through toxExamples:tox
tox -e py27 -- tests/cache/test_redis_backend.py
tox -e py27 -- tests/cache/test_redis_backend.py::RedisAdvanced_SerializedRaw_Test
tox -e py27 -- tests/cache/test_redis_backend.py::HstoreTestTests pass on the enclosedredis.conffile:/usr/local/Cellar/redis/3.0.7/redis-server ./redis-server--6379.confLicenseThis project is available under the same MIT license asdogpile.cache.