Package: redipy
redipyredipyis a Python library that provides a uniform interface to Redis-like
storage systems. It allows you to use the same Redis API with different backends
that implement the same functionality, such as:redipy.memory: A backend that runs inside the current process and stores
data in memory using Python data structures.redipy.redis: A backend that connects to an actual Redis instance and
delegates all operations to it.WarningThis library is still early in development andnot all redis functions are
available yet!
If you need certain functionality or found a bug, have a look at thecontributingsection.
It is easy to add redis functions to the API.Quick AccessInstallationUsageFeaturesCustom ScriptsSimple ExampleAdvanced ExampleLimitationsContributingIf You Find a BugMissing Redis or Lua FunctionsImplementing New Redis FunctionsChangelogLicenseFeedbackInstallationYou can installredipyusing pip:pipinstallredipy🔝UsageTo useredipy, you need to import the library and create aredipyclient
object with the desired backend. For example:# Import the redipy libraryimportredipy# Create a redipy client using the memory backendr=redipy.Redis()# Create a redipy client using the redis backendr=redipy.Redis(host="localhost",port=6379)# Or preferredr=redipy.Redis(cfg={"host":"localhost","port":6379,"passwd":"","prefix":"",})# You can specify the backend explicitly to ensure that the correct parameters# are passed to the constructorr=redipy.Redis(backend="redis",cfg={"host":"localhost","port":6379,"passwd":"","prefix":"",})Theredipyclient object supports similar methods and attributes to the
officialredisPython client library.
You can use them as you would normally do withredis. For example:# Set some valuesr.set("foo","bar")r.set("baz","qux")# Get some valuesr.get("foo")# "bar"r.get("baz")# "qux"# Push some valuesr.lpush("mylist","a","b","c")r.rpush("mylist","d")# Pop valuesr.lpop("mylist")# "c"r.rpop("mylist",3)# ["d", "a", "b"]🔝FeaturesThe main features ofredipyare:Flexibility: You can choose from different backends that suit your needs and
preferences, without changing your code or learning new APIs.Adaptability: You can start your project small with the memory backend and
only switch to a full redis server once the application grows.Scripting: You can create backend independent redis scripts without using lua.
Scripts are written using a symbolic API in python.Compatibility: You can use any Redis client or tool with any backend.Mockability: You can use redipy in tests that require redis with the memory
backend to easily mock the functionality without actually having to run a
redis server in the background. Also, this avoids issues that might occur
when running tests in parallel with an actual redis server.Performance: You can leverage the high performance of Redis or other backends
that offer fast and scalable data storage and retrieval.Migration: You can easily migrate data between different backends, or use
multiple backends simultaneously.🔝Custom ScriptsRedis scripts can be defined via a symbolic API in python and can be executed
by any backend.🔝Simple ExampleHere, we are writing a filter function that drains a redis list
and puts items into a "left" and a "right" list by comparing each items
numerical value with a givencmpvalue:importredipy# set up scriptctx=redipy.script.FnContext()# add argumentcmp=ctx.add_arg("cmp")# add key argumentsinp=redipy.script.RedisList(ctx.add_key("inp"))left=redipy.script.RedisList(ctx.add_key("left"))right=redipy.script.RedisList(ctx.add_key("right"))# add local variable which contains the current value pop'ed from the listcur=ctx.add_local(inp.lpop())# we consume "inp" until it is emptyloop=ctx.while_(cur.ne_(None))# push the value to the list depending on whether it is smaller than `cmp`b_then,b_else=loop.if_(redipy.script.ToNum(cur).lt_(cmp))b_then.add(left.rpush(cur))b_else.add(right.rpush(cur))# pop next value and store in local variableloop.add(cur.assign(inp.lpop()))# the script doesn't return a valuectx.set_return_value(None)# make sure to build the script only once and reuse the filter_list functionfilter_list=r.register_script(ctx)r.rpush("mylist","1","3","2","4")filter_list(keys={"inp":"mylist","left":"small","right":"big",},args={"cmp":3,})r.lpop("mylist",4)# []r.lpop("small",4)# ["1", "2"]r.lpop("big",4)# ["3", "4"]🔝Advanced ExampleHere, we are implementing and object stack with fall-through lookup. Each frame
in the stack has its own fields. If the user tries to access a field that
doesn't exist in the current stack frame (and they are usingget_cascading)
the accessor will recursively go down the stack until a value for the given
field is found (or the end of the stack is reached).fromtypingimportcastfromredipyimportRedisClientAPIfromredipy.scriptimport(ExecFunction,FnContext,JSONType,RedisHash,RedisVar,Strs,ToIntStr,ToNum,)classRStack:"""An example class that simulates a key value stack."""def__init__(self,rt:RedisClientAPI)->None:self._rt=rtself._set_value=self._set_value_script()self._get_value=self._get_value_script()self._pop_frame=self._pop_frame_script()self._get_cascading=self._get_cascading_script()defkey(self,base:str,name:str)->str:"""Compute the key.Args:base (str): The base key.name (str): The name.Returns:str: The key associated with the name."""returnf"{base}:{name}"defpush_frame(self,base:str)->None:"""Pushes a new stack frame.Args:base (str): The base key."""self._rt.incrby(self.key(base,"size"),1)defpop_frame(self,base:str)->dict[str,str]:"""Pops the current stack frame and returns its values.Args:base (str): The base key.Returns:dict[str, str] | None: The content of the stack frame."""res=self._pop_frame(keys={"size":self.key(base,"size"),"frame":self.key(base,"frame"),},args={})ifresisNone:return{}returncast(dict,res)defset_value(self,base:str,field:str,value:str)->None:"""Set a value in the current stack frame.Args:base (str): The base key.field (str): The field.value (str): The value."""self._set_value(keys={"size":self.key(base,"size"),"frame":self.key(base,"frame"),},args={"field":field,"value":value})defget_value(self,base:str,field:str)->JSONType:"""Returns a value from the current stack frame.Args:base (str): The base key.field (str): The field.Returns:JSONType: The value."""returnself._get_value(keys={"size":self.key(base,"size"),"frame":self.key(base,"frame"),},args={"field":field})defget_cascading(self,base:str,field:str)->JSONType:"""Returns a value from the stack. If the value is not in the currentstack frame the value is recursively retrieved from the previousstack frames.Args:base (str): The base key.field (str): The field.Returns:JSONType: The value."""returnself._get_cascading(keys={"size":self.key(base,"size"),"frame":self.key(base,"frame"),},args={"field":field})def_set_value_script(self)->ExecFunction:ctx=FnContext()rsize=RedisVar(ctx.add_key("size"))rframe=RedisHash(Strs(ctx.add_key("frame"),":",ToIntStr(rsize.get(default=0))))field=ctx.add_arg("field")value=ctx.add_arg("value")ctx.add(rframe.hset({field:value,}))ctx.set_return_value(None)returnself._rt.register_script(ctx)def_get_value_script(self)->ExecFunction:ctx=FnContext()rsize=RedisVar(ctx.add_key("size"))rframe=RedisHash(Strs(ctx.add_key("frame"),":",ToIntStr(rsize.get(default=0))))field=ctx.add_arg("field")ctx.set_return_value(rframe.hget(field))returnself._rt.register_script(ctx)def_pop_frame_script(self)->ExecFunction:ctx=FnContext()rsize=RedisVar(ctx.add_key("size"))rframe=RedisHash(Strs(ctx.add_key("frame"),":",ToIntStr(rsize.get(default=0))))lcl=ctx.add_local(rframe.hgetall())ctx.add(rframe.delete())b_then,b_else=ctx.if_(ToNum(rsize.get(default=0)).gt_(0))b_then.add(rsize.incrby(-1))b_else.add(rsize.delete())ctx.set_return_value(lcl)returnself._rt.register_script(ctx)def_get_cascading_script(self)->ExecFunction:ctx=FnContext()rsize=RedisVar(ctx.add_key("size"))base=ctx.add_local(ctx.add_key("frame"))field=ctx.add_arg("field")pos=ctx.add_local(ToNum(rsize.get(default=0)))res=ctx.add_local(None)cur=ctx.add_local(None)rframe=RedisHash(cur)loop=ctx.while_(res.eq_(None).and_(pos.ge_(0)))loop.add(cur.assign(Strs(base,":",ToIntStr(pos))))loop.add(res.assign(rframe.hget(field)))loop.add(pos.assign(pos-1))ctx.set_return_value(res)returnself._rt.register_script(ctx)🔝LimitationsThe current limitations ofredipyare:Not all Redis commands are supported yet: This will eventually be resolved.The API differs slightly: Most notably stored values are always strings
(i.e., the bytes returned by redis are decoded as utf-8).The semantic of redis functions inside scripts has been altered to feel more
natural coming from python: Redis functions inside lua scripts often differ
greatly from the documented behavior. For example,LPOPreturnsfalsefor
an empty list inside lua (instead ofnilorcjson.null). WhileLPOPreturnsNonein the python API. The script API ofredipyhas been altered
to match the python API more closely. As the user doesn't code in lua directly
the benefit of having a more consistent API outweighs the more complicated lua
code that needs to be generated in the backend.Scripts aim to use python semantics as best as possible: In lua array indices
start at 1. The script API uses a 0 based indexing system and transparently
adjusts indices in the lua backend. Other, similar changes are performed
as well.Scripts use JSON to pass arguments and return values: The arguments to the
script are passed as JSON bytes for the lua backend. Keys are passed as is.
The return value of the script is also converted into JSON when moving from
lua to python. Note, that the empty dictionary ({}) and the empty list
([]) are indistinguishable in lua soNoneis returned instead of setting
the return value to either of these.🔝ContributingAny contribution, even if it is just creating an issue for a bug,
is much appreciated.🔝If You Find a BugIf you encounter a bug, please open an issue to draw attention to it or give
a thumbsup if the issue already exists. This helps with prioritizing
implementation efforts. Even if you cannot solve the bug yourself,
investigating why it happens or creating a PR to add test cases helps a lot.
If you have a fix for a bug don't hesistate to open a PR.🔝Missing Redis or Lua FunctionsIf you encounter a missing redis or lua function please consider adding it
yourself (see theimplementingsection).
Here also opening an issue or giving a thumbsup to existing issues helps
with prioritization.However, if you need it only in your local setup
without API support or support for multiple backends, pipelines, etc. you can
use the raw underlying redis connection viaredipy.main.Redis.get_redis_runtimeandredipy.redis.conn.RedisConnection.get_connectionor make use of
the plug-in mechanism.For the memory backend you can useredipy.memory.rt.LocalRuntime.add_redis_function_pluginorredipy.memory.rt.LocalRuntime.add_general_function_plugin. The methods need
a module that contains subclasses ofredipy.plugin.LocalRedisFunctionandredipy.plugin.LocalGeneralFunctionrespectively. Once the new functions are
defined via loading the plugin they can be used in aredipy.script.FnContextviaredipy.script.RedisFnorredipy.script.CallFnrespectively.Note, thatredipy.script.RedisFnandredipy.script.CallFncan always be
used in redis backend scripts. However, calling functions this way will have
the native lua behavior which can lead to surprising results. To patch those
up as well you can useredipy.redis.lua.LuaBackend.add_redis_patch_plugin,redipy.redis.lua.LuaBackend.add_general_patch_plugin, andredipy.redis.lua.LuaBackend.add_helper_function_pluginto add the subclasses
ofredipy.plugin.LuaRedisPatch,redipy.plugin.LuaRedisPatch, andredipy.plugin.HelperFunctionrespectively. Those functions then can also be
used with theredipy.script.RedisFnandredipy.script.CallFncommands.Adding functions as described above is discouraged as it may lead to
inconsistent support of different backends and inconsistent behavior across
different backends.🔝Implementing New Redis FunctionsThe easiest way to contribute toredipyis to pick some redis API functions
that have not (or not completely) beenimplementedinredipyyet. It is also much appreciated if you just add test cases or the stubs in a
PR. For a full implementation follow these steps:Add the signature of the function toredipy.api.RedisAPI. Adjust as
necessary from the redis spec to get a pythonic feel. Also, add the signature
toredipy.api.PipelineAPIbut withNoneas return value. Additionally,
add the redirect to the backend inredipy.main.Redis.Implement the function inredipy.redis.conn.RedisConnectionandredipy.redis.conn.PipelineConnection. This should
be straightforward as there are not too many changes expected. Don't forget
to convert bytes into strings via...decode("utf-8")(there are various
helper functions for this inredipy.util).Add tests totest/test_sanity.pyto determine the function's behavior in
lua (especially its edge cases).If the lua behavior needs to be changed to provide a better feel you can add
a monkeypatch for the function call by either creating a class inredipy.redis.rpatchto directly change the returned expr for the execution
graph or using a lua helper function via adding a class toredipy.redis.helpers(you need to use a patch to use the helper in the
right location).Next, add and implement the functionality inredipy.memory.state.Machineand add the appropriate redirects inredipy.memory.rt.LocalRuntimeandredipy.memory.rt.LocalPipeline.To make the new function accessible in scripts from the memory backend add
a class inredipy.memory.rfun.Add the approriate class or method in the rightredipy.symbolic.r...pyfile. If it is a new class / file add an import toredipy.script.Add a new test intest/test_api.pyto verify the new function works inside
a script for all backends. You can runmake pytest FILE=test/test_api.pyto execute the test andmake coverage-reportto verify that the new code
is executed.Make suremake lint-allpasses, as well as, all tests (make pytest)
run without issue.You can submit your patch as pull requesthere.🔝ChangelogThe changelog can be foundhere.🔝Licenseredipyis licensed under theApache License (Version 2.0).🔝FeedbackIf you have any questions, suggestions, or issues withredipy, please feel
free toopen an issueon GitHub. I would love to hear your feedback
and improveredipy. Thank you!🔝